Image processing method and image processing device

ABSTRACT

The image processing method includes a luminance value information obtaining step of obtaining effective radiance values from a subject, and an image generating step of generating a picture image as a set of unit regions each of which has a luminance value obtained by at least partially removing a regular reflection light component on a surface of the subject from the effective radiance values.

TECHNICAL FIELD

The following disclosure relates to an image processing method, an imageprocessing device, and an image processing program for carrying out aprocess with respect to a picture image that is used in irisauthentication.

BACKGROUND ART

In recent years, a technique of separating, in a picture image of asubject, a diffuse reflection component from a specular reflectioncomponent in reflected light from the subject has been developed. Thisis because many of image processing algorithms are predicated on diffusereflection, and specular reflection may cause decrease in performance ofimage processing algorithms. Patent Literature 1 discloses an example ofa technique to separate the above two reflection components.

Patent Literature 1 discloses an image processing method including thefollowing steps (1) through (5).

(1) Under arbitrary illumination, a plurality of picture images of asubject are taken with a camera through polarizing elements whoseprincipal axis directions are different from each other.

(2) For each of pixels in a pixel group in which specular reflectionoccurs in the plurality of picture images, an incidence plane isidentified based on a normal vector and an eye vector of the subject.

(3) For each of the pixels, an incident angle is identified based on thenormal vector and the eye vector of the subject.

(4) A pixel set is formed by clustering pixels which have similarincidence planes and also similar incident angles.

(5) In the pixel set, stochastic independence between a diffusereflection component and a specular reflection component is presumed,and those reflection components are separated from each other.

According to the image processing method disclosed in Patent Literature1, the diffuse reflection component can be separated from the specularreflection component even under a general illumination environment.

CITATION LIST Patent Literature

[Patent Literature 1]

Japanese Patent No. 3955616 (Registered on May 11, 2007)

SUMMARY OF INVENTION Technical Problem

However, Patent Literature 1 does not disclose that iris authenticationis carried out while reducing a regular reflection light componentcontained in reflected light from the subject.

An object of an aspect of the present disclosure is to provide an imageprocessing method and the like which can reduce a regular reflectionlight component.

Solution to Problem

In order to attain the object, the image processing method in accordancewith an aspect of the present disclosure includes: a luminance valueinformation obtaining step of obtaining effective radiance values of asubject by an image pickup device in which pixel units aretwo-dimensionally arranged, each of the pixel units including aplurality of pixels which are associated with respective of a pluralityof polarizing elements whose principal axis directions are differentfrom each other, and each of the effective radiance values being aneffective radiance value in the image pickup device; and an imagegenerating step of generating a picture image including an image of thesubject with use of the effective radiance values obtained from thesubject, in the image generating step, a luminance value being obtainedby at least partially removing a regular reflection light component on asurface of the subject from the effective radiance values of theplurality of pixels included in each of the pixel units corresponding toat least part of the subject, and the picture image being generated as aset of unit regions each of which has the luminance value.

Moreover, the image processing method in accordance with an aspect ofthe present disclosure includes the steps of: obtaining a picture imageof a subject taken by an image pickup device in which pixel units aretwo-dimensionally arranged, each of the pixel units including aplurality of pixels which are associated with respective of a pluralityof polarizing elements whose principal axis directions are differentfrom each other; and calculating, with use of an output from the imagepickup device, a luminance distribution of S-polarized light, theluminance distribution depending on an incident angle with respect tothe subject, the incident angle being determined based on a position onthe subject which position corresponds to a two-dimensional position ofeach of the pixel units in the image pickup device.

Moreover, the image processing device in accordance with an aspect ofthe present disclosure includes: an image pickup device in which pixelunits are two-dimensionally arranged, each of the pixel units includinga plurality of pixels which are associated with respective of aplurality of polarizing elements whose principal axis directions aredifferent from each other; a luminance value information obtainingsection which obtains effective radiance values of a subject by theimage pickup device, each of the effective radiance values being aneffective radiance value in the image pickup device; and an imagegenerating section which generates a picture image including an image ofthe subject with use of the effective radiance values obtained from thesubject, the image generating section obtaining a luminance value by atleast partially removing a regular reflection light component on asurface of the subject from effective radiance values of the pluralityof pixels included in each of the pixel units corresponding to at leastpart of the subject, and generates the picture image as a set of unitregions each of which has the luminance value.

Advantageous Effects of Invention

According to the image processing method and the image processing devicein accordance with aspects of the present disclosure, it is possible toprovide an image processing method and the like which can reduce aregular reflection light component.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart showing an example of a flow of processes in animage processing method in accordance with Embodiment 1.

FIG. 2 is a view for explaining an overview of an image processingdevice.

FIG. 3 is a view for explaining an overview of image processing carriedout by an image processing device.

FIG. 4 is a block diagram illustrating a configuration of a personaldigital assistant which includes the image processing device inaccordance with Embodiment 1.

Each of (a) and (b) of FIG. 5 is a view illustrating another example ofan integrated polarizer included in a camera.

FIG. 6 is a view for explaining values for use in processes in the imageprocessing device.

FIG. 7 is a graph showing a received-light intensity of light receivedby a pixel, with respect to a polarization angle.

FIG. 8 is a graph showing reflection coefficients of S-polarized lightand P-polarized light with respect to an incident angle.

FIG. 9 is a flowchart showing another example of a flow of processescarried out in the image processing method in accordance with Embodiment1.

Each of (a) and (b) of FIG. 10 is a flowchart showing still anotherexample of a flow of processes carried out in the image processingmethod in accordance with Embodiment 1.

FIG. 11 is a graph showing a reflection coefficient of S-polarized lightwith respect to an incident angle of light to a rubber ball.

FIG. 12 is a view showing an overview of a picture image showing anexperimental result of image processing carried out by the imageprocessing device in accordance with Embodiment 1.

FIG. 13 is a block diagram illustrating a configuration of an imageprocessing device in accordance with Embodiment 3.

FIG. 14 is a flowchart showing a flow of processes carried out by theimage processing device in accordance with Embodiment 3.

FIG. 15 is a block diagram illustrating a configuration of an electronicinformation apparatus including an image processing device in accordancewith Embodiment 4.

FIG. 16 is a flowchart showing a flow of processes in the imageprocessing device in accordance with Embodiment 4.

FIG. 17 is a graph showing an example of luminance values of pixels in aunit region.

FIG. 18 is a block diagram illustrating a configuration of an electronicinformation apparatus including an image processing device in accordancewith Embodiment 5.

FIG. 19 is a flowchart showing a flow of processes in the imageprocessing device in accordance with Embodiment 5.

FIG. 20 is a graph showing an example of luminance values of pixels in aunit region.

FIG. 21 is a graph showing parameters included in a formula (5-1).

(a) of FIG. 22 is a view showing an example of a look-up table, (b) ofFIG. 22 is a view showing an example of a table prepared by a minimumluminance value estimating section, (c) of FIG. 22 is a view showing anexample of signal intensities of luminance values, and (d) of FIG. 22 isa graph showing an example of the signal intensities shown in (c) ofFIG. 22 and an example of signal intensities indicating luminance valuesestimated by the minimum luminance value estimating section.

FIG. 23 is a block diagram illustrating a configuration of an electronicinformation apparatus including an image processing device in accordancewith Embodiment 6.

FIG. 24 is a flowchart showing a flow of processes in the imageprocessing device in accordance with Embodiment 6.

FIG. 25 is graphs showing examples of effective radiance values ofpixels included in a pixel unit, estimated values of possible effectiveradiance values of the pixels included in the pixel unit, and estimatedvalues of a diffusion light component. (a) of FIG. 25 is a graph of apixel unit which reflected light enters in a case where light hasreached an eyeball at an incident angle of 30° and has been reflected bythe eyeball, (b) of FIG. 25 is a graph of a pixel unit which reflectedlight enters in a case where light has reached an eyeball at an incidentangle of 20° and has been reflected by the eyeball, and (c) of FIG. 25is a graph of a pixel unit which reflected light enters in a case wherelight has reached an eyeball at an incident angle of 10° and has beenreflected by the eyeball.

FIG. 26 is a block diagram illustrating a configuration of an electronicinformation apparatus including an image processing device in accordancewith Embodiment 8.

FIG. 27 is a flowchart showing a flow of processes in the imageprocessing device in accordance with Embodiment 8.

FIG. 28 is a view showing an arrangement of pixel units. (a) of FIG. 28is a view illustrating a state in which pixel units each of which ismade up of nine pixels are arranged so as not to overlap with eachother, (b) of FIG. 28 is a view illustrating a state in which pixelunits each of which is made up of four pixels are arranged so as not tooverlap with each other, (c) of FIG. 28 is a view illustrating a statein which pixel units each of which is made up of nine pixels arearranged so as to partially overlap with each other, and (d) of FIG. 28is a view illustrating a state in which pixel units each of which ismade up of four pixels are arranged so as to partially overlap with eachother.

FIG. 29 is a flowchart showing a flow of processes in an imageprocessing device which simultaneously executes an image processing ofEmbodiment 6 and another image processing.

FIG. 30 is a view for explaining a case where a subject includeseyeballs of both eyes.

DESCRIPTION OF EMBODIMENTS

[Embodiment 1]

The following description will discuss details of Embodiment 1 of thepresent invention. An image processing device in accordance withEmbodiment 1 is a device which carries out authentication based on apicture image of an iris of an eyeball of a human.

(Overview of Image Processing Device 10)

First, the following description will discuss an overview of an imageprocessing device 10. FIG. 2 is a view for explaining an overview of theimage processing device 10. The image processing device 10 carries outimage processing with respect to a picture image of a subject.Specifically, the image processing device 10 carries out a process ofseparating a diffuse reflection component from a specular reflectioncomponent which are contained in reflected light from the subject. InEmbodiment 1, the image processing device 10 is mounted on a personaldigital assistant 1.

The personal digital assistant 1 is, for example, a terminal which canseparate the above two reflection components from each other in apicture image of an eyeball E (subject) of a user and carry out irisauthentication of the user with use of the picture image in which one ofthe two reflection components are removed. The personal digitalassistant 1 includes the image processing device 10 and a camera 20 asillustrated in FIG. 2. Details of the image processing device 10 will bedescribed later. Note that the subject in accordance with an aspect ofthe present invention is not limited to an eyeball, provided that animage may be reflected on the subject.

The camera 20 takes a picture image of the subject in accordance with auser operation. In Embodiment 1, as illustrated in FIG. 2, the camera 20takes a picture image of an eyeball E of the user. Moreover, the camera20 mainly includes an integrated polarizer 21 and a light-receivingelement 22 (image pickup device) (see FIG. 3). The integrated polarizer21 and the light-receiving element 22 are laminated in this order in adirection in which light enters the camera 20.

The integrated polarizer 21 is made up of a plurality of polarizingelements whose principal axis directions are different from each other.In the integrated polarizer 21 of Embodiment 1, the plurality ofpolarizing elements correspond to respective pixels. In Embodiment 1,the integrated polarizer 21 includes nine polarizing elements, i.e.,polarizing elements 21 a through 21 i which correspond to respectivenine pixels which are adjacent to each other (see FIG. 2). Specifically,the polarizing elements 21 a through 21 i have, in the respectivepixels, polarization angles of 0°, 20°, 40°, 60°, 80°, 100°, 120°, 140°,and 160°, respectively.

The light-receiving element 22 has a configuration in which pixel units,each of which is made up of the plurality of pixels associated with therespective polarizing elements 21 a through 21 i, are two-dimensionallyarranged.

In a case where iris authentication of the user is carried out with thepersonal digital assistant 1, a picture image of the eyeball E of theuser is taken by the camera 20. As illustrated in FIG. 2, in a casewhere the eyeball E of the user is irradiated with outside light orindoor light, the light is reflected on a surface of the eyeball E, andthus reflected light Lr is to enter the camera 20.

In the case where the eyeball E of the user is irradiated with outsidelight (sunlight) or indoor light and the camera 20 obtains the reflectedlight Lr which is the outside light or indoor light reflected on theiris, the camera 20 obtains a picture image including an image of theiris of the user, and the personal digital assistant 1 carries out userauthentication by analyzing the image of the iris. Meanwhile, in a casewhere an object O exists in sight of the user, an image of the object Ois reflected on the eyeball E due to influence of outside light orindoor light, and a reflected image Ir is formed on the eyeball E (seeFIG. 2). That is, in a case where the object O reflects outside light orindoor light and the eyeball E is irradiated with the light thusreflected by the object O, the reflected image Ir is formed on theeyeball E. Then, the camera 20 obtains, as the reflected image Ir, thereflected light Lr (i.e., outside light or indoor light which has beenreflected), and thus obtains a picture image including the reflectedimage Ir. Unless the personal digital assistant 1 carries out a processof removing the reflected image Ir from the obtained picture image thatincludes the image of the iris and the reflected image Ir, there is apossibility that the image analysis on the iris is influenced by thereflected image Ir, and accurate iris authentication cannot be carriedout.

In particular, under irradiation with sunlight, an image is clearlyreflected on the eyeball E of the user, and it is therefore difficult tocarry out accurate iris authentication outdoors. Although it is possibleto reduce influence of sunlight in iris authentication by irradiatingthe eyeball E of the user with light having an intensity higher thanthat of sunlight, such irradiation of the eyeball E or skin with thehighly intense light may cause deterioration in state of the eyeball Eor the skin.

The personal digital assistant 1 in accordance with Embodiment 1includes the integrated polarizer 21 and the image processing device 10,and is therefore possible to carry out accurate iris authenticationwhile reducing influence of the reflected image Ir in iris imageanalysis, without irradiating the eyeball E with the highly intenselight as above described.

Next, the following description will discuss an overview of a processthat is carried out by the image processing device 10 for reducing theabove described influence, with reference to FIG. 3. FIG. 3 is a viewfor explaining an overview of an image processing that is carried out bythe image processing device 10. Note that, in FIG. 3, the integratedpolarizer 21 is illustrated in a simplified manner.

In an example shown in FIG. 3, a picture image of the eyeball E of theuser is taken for iris authentication under irradiation with sunlight.Moreover, a reflected image Ir of an object O is formed on the eyeball Eof the user.

In a case where a picture image of the eyeball E of the user is takenwith the camera 20 for iris authentication, reflected light Lr from theeyeball E of the user is received by the light-receiving element 22 viathe integrated polarizer 21.

Here, in general, an intensity of light (in this case, reflected lightLr showing the iris used in an authentication process) forming an imageused in an image processing is mostly based on a diffuse reflectioncomponent. In Embodiment 1, the light is processed as light showingsurface information which is indicative of a surface of the eyeball E(specifically, iris) and is necessary for the authentication process. Onthe other hand, an intensity of light (in this case, reflected light Lrshowing the object O that causes adverse influence on the authenticationprocess) forming an image that is a noise to be removed in the imageprocessing is mostly based on a specular reflection component. Thespecular reflection component includes an S-polarized light sourcecomponent (S-polarized light component) and a P-polarized light sourcecomponent (P-polarized light component).

In FIG. 3, a picture image of the eyeball E including the reflectedimage Ir and the iris is taken, and therefore reflected light Lrcontaining the above three types of reflection components is received bythe light-receiving element 22 via the integrated polarizer 21. In thiscase, it is necessary to remove the S-polarized light component and theP-polarized light component from the reflected light Lr which has beenreceived by the light-receiving element 22.

In Embodiment 1, a digital conversion process is carried out withrespect to the reflected light Lr which has been received by thelight-receiving element 22, and a process (S-wave/P-wave removal) ofremoving the S-polarized light component and the P-polarized lightcomponent is carried out with respect to the reflected light Lr whichhas been converted into a digital signal. Thus, the reflected image Iris removed from the picture image of the eyeball E which is necessaryfor the authentication process (see FIG. 3). The personal digitalassistant 1 encodes the picture image from which the reflected image Irhas been removed (i.e., the picture image including only the image ofthe iris of the eyeball E), and thus generates data of the eyeball E forauthentication. As such, the personal digital assistant 1 can carry outaccurate iris authentication.

Note that, in order to carry out iris authentication more accurately, itis possible to carry out a known independent component analysis (ICA)process after the process of removing the S-polarized light componentand the P-polarized light component.

(Configuration of Personal Digital Assistant 1)

FIG. 4 is a block diagram illustrating a configuration of the personaldigital assistant 1 which includes the image processing device 10 forcarrying out the image processing method of Embodiment 1. As illustratedin FIG. 4, the personal digital assistant 1 includes the imageprocessing device 10, the camera 20, and a distance measuring device 30which (i) measures a distance from the camera 20 to each point on thesurface of the eyeball E of the user and (ii) transmits the distance toan S-polarized light calculating section 12 (described later).

In Embodiment 1, an example is described in which the personal digitalassistant 1 integrally includes the image processing device 10, thecamera 20, and the distance measuring device 30. Note, however, thatthose constituent members do not need to be integrally provided. Thatis, it is only necessary that the image processing device 10 can obtaina picture image taken with use of the camera 20 which is separated fromthe image processing device 10 and can obtain the distance measured bythe distance measuring device 30 which is separated from the imageprocessing device 10.

In Embodiment 1, pixels of the camera 20 are constituted by chargecoupled devices (CCD). Alternatively, pixels of the camera 20 can beconstituted by complementary metal oxide semiconductors (CMOS).

Each of (a) and (b) of FIG. 5 is a view illustrating another example ofthe integrated polarizer included in the camera 20. As described withreference to FIG. 2, the camera 20 in accordance with Embodiment 1includes the integrated polarizer 21 which have the nine polarizingelements, i.e., the polarizing elements 21 a through 21 i. Note,however, that a camera used in an aspect of the present invention caninclude, for example, an integrated polarizer 21A which has fourpolarizing elements, i.e., polarizing elements 21 j, 21 k, 21 l, and 21m as illustrated in (a) of FIG. 5. Alternatively, a camera used in anaspect of the present invention can include, for example, an integratedpolarizer 21B which has two types of polarizing elements, i.e.,polarizing elements 21 n and 210 as illustrated in (b) of FIG. 5.

(Image Processing Device 10)

The image processing device 10 includes an iris detecting section 11, anS-polarized light calculating section 12, a P-polarized lightcalculating section 13, a diffusion light calculating section 14, and anauthenticating section 15.

The iris detecting section 11 obtains a picture image taken by thecamera 20, and identifies a region corresponding to an iris of the userin the picture image. A process that is carried out by the irisdetecting section 11 is known in the field of, for example,authentication using a picture image of iris, and is therefore notdescribed in this specification.

The S-polarized light calculating section 12 calculates luminancedistribution of S-polarized light contained in the picture image. Aprocess carried out by the S-polarized light calculating section 12 willbe described later.

The P-polarized light calculating section 13 calculates luminancedistribution of P-polarized light contained in the picture image. TheP-polarized light calculating section 13 calculates the luminancedistribution of the P-polarized light based on the luminancedistribution of the S-polarized light and Fresnel's law.

The diffusion light calculating section 14 calculates luminancedistribution of diffusion light contained in the picture image.Specifically, the diffusion light calculating section 14 subtracts, froma luminance value of each pixel, luminance values of the S-polarizedlight and the P-polarized light in that pixel. By the subtractionprocess, it is possible to obtain a picture image from which theS-polarized light component and the P-polarized light component havebeen removed and which contains only a diffusion light component.

The authenticating section 15 carries out user authentication with useof an iris image included in the picture image containing only thediffusion light component. The iris authentication carried out by theauthenticating section 15 is a known technique, and is therefore notdescribed in this specification.

(Process of S-Polarized Light Calculating Section 12)

FIG. 6 is a view for explaining values used in a process carried out bythe image processing device. In the descriptions below, numerical valuesare defined as follows.

-   -   R: Distance from lens of camera 20 to center of eyeball E    -   r: Radius of eyeball E    -   θ: Incident angle of light to eyeball E (angle formed by (i)        straight line L1 (first virtual line) connecting position P on        eyeball at which position P light enters with center of eyeball        and (ii) straight line L2 (second virtual line) connecting that        position P with center of lens of camera 20)    -   ϕ: Angle formed by (i) straight line connecting center of lens        of camera 20 with center of eyeball E and (ii) the straight line        L1

The S-polarized light calculating section 12 calculates, with use ofluminance values (outputs) of pixels of the light-receiving element 22,a luminance distribution of S-polarized light which depends on anincident angle θ that is determined based on a position P on the eyeballE which position P corresponds to a two-dimensional position of a pixelunit. The following description will discuss a process carried out bythe S-polarized light calculating section 12.

(First Process of S-Polarized Light Calculating Section 12)

First, the S-polarized light calculating section 12 identifies a pixelin the light-receiving element 22 which pixel corresponds to a point(hereinafter, referred to as “Brewster point” according to need) on theeyeball at which point an incident angle θ becomes a Brewster angle.Specifically, the S-polarized light calculating section 12 calculates anangle ϕ based on a formula (1-1) below. A formula (1-2) is obtained bymodifying the formula (1-1).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{R\mspace{11mu}\sin\;{\phi\left( {{R\mspace{11mu}\cos\;\phi} - r} \right)}\tan\;\theta} & \left( {1\text{-}1} \right) \\\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{{\therefore{\tan\;\theta}} = \frac{R\mspace{11mu}\sin\;\phi}{{R\mspace{11mu}\cos\;\phi} - r}} & \left( {1\text{-}2} \right)\end{matrix}$

In the formulae (1-1) and (1-2), R is measured by the distance measuringdevice 30. More accurately, the distance measuring device 30 measures adistance from the lens of the camera 20 to the surface of the eyeball E,that is, a distance obtained by subtracting r from R. In a case of aneyeball of a human, a value of r is substantially constant, i.e., 7.4mm. Therefore, it is possible to calculate R from measurement dataobtained by the distance measuring device 30. Further, a refractiveindex n of a cornea of an eyeball is 1.376, and accordingly the Brewsterangle (=θ) is assumed to be 53.1° in Embodiment 1. Based on thosenumerical values, the S-polarized light calculating section 12 cancalculate the angle ϕ by using the formula (1-1) or (1-2).

Here, a pixel which exists on the straight line connecting the camera 20with the center of the eyeball E and corresponds to the surface of theeyeball E is a pixel which is nearest to the camera 20 in a region ofthe eyeball E. Based on the position of the pixel and the distance, itis possible to identify a pixel which corresponds to the Brewster pointin the picture image of the eyeball E. Note that a plurality of pixelsare identified in the picture image as pixels each corresponding to theBrewster point.

Next, the S-polarized light calculating section 12 identifies a pixelunit which is included in the light-receiving element 22 of the camera20 and includes identified pixels. The pixel unit is a group of pixelswhich are associated with the respective plurality of polarizingelements. In Embodiment 1, the pixel unit is associated with ninepolarizing elements whose principal axis directions are different fromeach other. The pixel units are two-dimensionally arranged in thelight-receiving element 22 of the camera 20. In FIG. 6, light which hasentered the position P is received at a two-dimensional position of apixel unit of the light-receiving element 22 which pixel unit is on thestraight line L2 connecting the position P on the eyeball with thecenter of the lens of the camera 20.

Next, the S-polarized light calculating section 12 subtracts a minimumluminance value from a maximum luminance value of pixels included in theidentified pixel unit, and thus calculates a luminance value ofS-polarized light at the Brewster point. In this case, for example, theluminance value of the S-polarized light at the Brewster point can be anaverage value of differences obtained by subtracting minimum luminancevalues from respective maximum luminance values of pixels in all ofidentified pixel units. Alternatively, the luminance value of theS-polarized light at the Brewster point can be a difference calculatedby subtracting a minimum luminance value from a maximum luminance valueof pixels in only an arbitrary pixel unit.

FIG. 7 is a graph showing a received-light intensity of light receivedby a pixel, with respect to a polarization angle (i.e., an angle in ofthe principal axis direction). In the graph shown in FIG. 7, ahorizontal axis represents the polarization angle, and a vertical axisrepresents the received-light intensity. The received-light intensitycontains a diffuse reflection component (k-term) indicative of an irisimage necessary for the authentication process and a specular reflectioncomponent indicative of a reflected image Ir which is obtained byoutside light and is a noise in the authentication process. As abovedescribed, the specular reflection component contains an S-polarizedlight component (f-term) and a P-polarized light component (g-term). TheS-polarized light component forms a substantially cosine function(cos(θ+Ψ) (Ψ is an arbitrary constant)) or a substantially sinefunction.

The diffuse reflection component is constant, regardless of thepolarization angle. Therefore, in the graph shown in FIG. 7, variationin received-light intensity with respect to the polarization angle isvariation in the specular reflection component, in particular, in theS-polarized light component.

FIG. 8 is a graph showing reflection coefficients (Fresnel's law) ofS-polarized light and P-polarized light with respect to the incidentangle θ. In the graph shown in FIG. 8, a horizontal axis represents anincident angle θ (see FIG. 6) of light to the eyeball E, and a verticalaxis represents a reflection coefficient. The reflection coefficient isa relative value indicating the reflection intensity. The reflectionintensity of the S-polarized light in the eyeball E corresponds to areceived-light intensity in a pixel, and the reflection intensity of theP-polarized light in the eyeball E corresponds to the received-lightintensity in the pixel. Formulae (2-1) and (2-2) below show relationsbetween an incident angle θ of light to the eyeball E and reflectioncoefficients of S-polarized light and P-polarized light contained inreflected light. In the formulae (2-1) and (2-2), n=1.376 as abovedescribed.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack & \; \\{{f(r)} = \left( \frac{{\cos\;\theta\;(r)} - \sqrt{n^{2} - {\sin^{2}{\theta(r)}}}}{{\cos\;\theta\;(r)} + \sqrt{n^{2} - {\sin^{2}{\theta(r)}}}} \right)^{2}} & \left( {2\text{-}1} \right) \\\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\{{g(r)} = \left( \frac{{n^{2}\cos\;\theta\;(r)} - \sqrt{n^{2} - {\sin^{2}{\theta(r)}}}}{{n^{2}\cos\;\theta\;(r)} + \sqrt{n^{2} - {\sin^{2}{\theta(r)}}}} \right)^{2}} & \left( {2\text{-}2} \right)\end{matrix}$

As shown in FIG. 8, the reflection coefficient of the P-polarized lightis substantially zero in a case where the incident angle θ is in thevicinity of the Brewster angle. That is, in pixels included in a pixelunit which includes a pixel corresponding to the Brewster point,variation in received-light intensity with respect to the polarizationangle can be considered as variation in S-polarized light.

The intensity of S-polarized light is known to sine-functionally varywith respect to variation in polarization angle. Therefore, bysubtracting a minimum luminance value from a maximum luminance value ofpixels included in the identified pixel unit, it is possible to obtain areflection intensity (amplitude) of S-polarized light in the pixel unit.

Subsequently, the S-polarized light calculating section 12 calculates areflection intensity of S-polarized light in pixel units other than theidentified pixel unit based on (i) the reflection coefficient and thereflection intensity of the S-polarized light with respect to theBrewster angle and (ii) a reflection coefficient of S-polarized light inthe pixel units other than the pixel unit corresponding to the Brewsterpoint. Specifically, the S-polarized light calculating section 12calculates a ratio between the reflection coefficient and the reflectionintensity of the S-polarized light with respect to the Brewster angle,and calculates, with use of the ratio, a reflection intensitycorresponding to a reflection coefficient with respect to an incidentangle θ other than the Brewster angle. That is, the S-polarized lightcalculating section 12 calculates a luminance value of S-polarized lightwith respect to an incident angle θ other than the Brewster angle, onthe basis of the luminance value of S-polarized light in the pixel inwhich the incident angle θ is the Brewster angle.

Note that, in a case where no outside light is reflected in the regionof the eyeball E in the picture image, the luminance value of each pixeldoes not contain a specular reflection component. In this case, each ofluminance values of the respective pixels in the pixel unit does notcontain an S-polarized light component, and does therefore notsine-functionally vary with respect to the polarization angle.

In view of this, the image processing device in accordance with anaspect of the present invention can further include a reflected imagedetermining section for determining, before the process of theS-polarized light calculating section 12, whether or not luminancevalues of respective pixels in a pixel unit sine-functionally vary withrespect to the polarization angle. A pixel unit to be determined by thereflected image determining section can be arbitrarily set by amanufacturer of the image processing device.

In a case where luminance values of respective pixels in the pixel unitdo not sine-functionally vary with respect to the polarization angle, itis possible to determine that no outside light is reflected on theeyeball E. In this case, it is possible that the image processing device10 does not calculate S-polarized light and P-polarized light, andcarries out authentication by the authenticating section 15 whileregarding a minimum luminance value in pixels included in each of pixelunits as a luminance value of a diffuse reflection component.

(Second Process of S-Polarized Light Calculating Section 12)

The above described pixel corresponding to the Brewster point is mostlypositioned in a circumference of the iris region in the picture image.Therefore, the pixel corresponding to the Brewster point is sometimesnot included in the iris region depending on a degree to which an eye ofthe user opens or on an angle at which the picture image is taken.

In such a case, the first process cannot be normally carried out, andtherefore the authentication by the iris fails. In a case where theauthentication based on the iris picture image generated by the firstprocess has failed, the S-polarized light calculating section 12 carriesout a second process (described below) while determining that the pixelcorresponding to the Brewster angle is not included in the eyeball Eregion in the light-receiving element 22.

In the second process, a luminance value of S-polarized light iscalculated by subtracting a minimum luminance value from a maximumluminance value of pixels included in a pixel unit, and this calculationis repeated for each of a plurality of pixel units included in thelight-receiving element 22. Thus, a luminance value of S-polarized lightin each of identified pixel units is calculated by subtracting a minimumluminance value from a maximum luminance value of pixels included inthat pixel unit. Subsequently, the S-polarized light calculating section12 carries out fitting, with use of the formula (2-1), on a relationbetween an incident angle θ corresponding to each pixel unit and aluminance value of S-polarized light, and thus calculates a value of n.

Note that, in the image processing device in accordance with Embodiment1, the S-polarized light calculating section 12 can carry out either oneof the first process and the second process first. Alternatively, theS-polarized light calculating section 12 may carry out only one of thefirst process and the second process.

(Process of P-Polarized Light Calculating Section 13)

As seen in FIG. 8 and the formulae (2-1) and (2-2), an intensity ratiobetween S-polarized light and P-polarized light in accordance withincident angles θ is determined based on Fresnel's law. Therefore, in acase where the S-polarized light calculating section 12 has carried outthe above described first process, the P-polarized light calculatingsection 13 can calculate a luminance distribution of P-polarized lightbased on, as above described, a luminance distribution of S-polarizedlight and Fresnel's law.

Moreover, in a case where the S-polarized light calculating section 12has carried out the above described second process, the value of n hasbeen calculated. Therefore, the P-polarized light calculating section 13can calculate an intensity of P-polarized light based on the abovedescribed formula (2-2).

(First Process of Image Processing Device 10)

FIG. 1 is a flowchart showing an example of a flow of processes in theimage processing method in accordance with Embodiment 1.

First, the camera 20 takes a picture image including the eyeball E ofthe user, and the distance measuring device 30 measures a distance fromthe camera 20 to the eyeball E (S1). The image processing device 10obtains data indicating the picture image and the distance. Then, theiris detecting section 11 executes, with respect to the picture imagethus taken, a process of detecting an iris region (S2).

Next, the S-polarized light calculating section 12 calculates aluminance value of S-polarized light at a point at which an incidentangle θ of light reflected toward the camera 20 is the Brewster angle inthe detected iris region (SA1). Further, the S-polarized lightcalculating section 12 calculates, based on the luminance value ofS-polarized light calculated in the step SA1 and Fresnel's law, aluminance value (luminance distribution) of S-polarized light at a pointat which an incident angle θ of light reflected toward the camera 20 isnot the Brewster angle (SA2).

Subsequently, the P-polarized light calculating section 13 calculates aluminance value (luminance distribution) of P-polarized light based onthe luminance distribution of S-polarized light calculated in the stepsSA1 and SA2 and on Fresnel's law (SA3). Further, the diffusion lightcalculating section 14 calculates luminance distribution of diffusionlight based on the luminance distribution of S-polarized light and theluminance distribution of P-polarized light (SA4).

The authenticating section 15 carries out authentication with use of apicture image (sometimes referred to as a picture image of diffusionlight) which shows the luminance distribution of diffusion light whichluminance distribution has been obtained in the step SA4 (SA5). Afterthat, the authenticating section 15 determines whether or not the usercan be authenticated based on the picture image of diffusion light (S3).In a case where the user has been authenticated (Y in S3), the imageprocessing device 10 ends the image processing.

On the other hand, in a case where the user has not been authenticated(N in S3), the image processing device 10 attempts authentication of theuser again through processes different from the steps SA1 through SA5,in order to improve accuracy in authentication. Specifically, first, theS-polarized light calculating section 12 calculates a luminance value ofS-polarized light for each of pixel units in the detected iris region(SB1). Next, the P-polarized light calculating section 13 calculates aluminance value of P-polarized light based on the luminance value ofS-polarized light calculated in the step SB1 (SB2). Then, the diffusionlight calculating section 14 calculates a luminance value of diffusionlight based on the luminance value of S-polarized light and theluminance value of P-polarized light (SB3). The authenticating section15 carries out authentication with use of a picture image of diffusionlight obtained in the step SB3 (SB4). Note that, in a case where theauthentication has failed in the step SB4, the authenticating section 15notifies the user of the failure of authentication via, for example, adisplay part (not illustrated) or the like included in the personaldigital assistant 1.

Note that, among the processes in the flowchart shown in FIG. 1, theprocesses of the steps SA1 through SA4 correspond to the above describedfirst process carried out by the S-polarized light calculating section12. Moreover, the processes of the steps SB1 through SB3 correspond tothe above described second process carried out by the S-polarized lightcalculating section 12.

(Second Process of Image Processing Device 10)

FIG. 9 is a flowchart showing another example of a flow of processescarried out in an image processing method in accordance withEmbodiment 1. According to the processes shown in FIG. 9, it is possibleto expect that authentication of the user can be carried out morequickly than the processes shown in FIG. 1.

In the processes shown in FIG. 9, first, the steps S1 and S2 areexecuted, as with the example shown in FIG. 1. Next, the imageprocessing device 10 executes authentication through the steps SB1through SB4, before the step SA1. Subsequently, the authenticatingsection 15 determines whether or not the user has been authenticated(S3). In a case where the user has not been authenticated (N in S3), theimage processing device 10 executes authentication by the steps SA1through SA5.

The authentication by the steps SB1 through SB4 can be carried out in ashorter time than the authentication by the steps SA1 through SA5.Therefore, according to the processes shown in FIG. 9, in a case wherethe authentication of the user can be carried out by the steps SB1through SB4, it is possible to authenticate the user more quickly thanthe processes shown in FIG. 1.

(Third Process of Image Processing Device 10)

(a) of FIG. 10 is a flowchart showing still another example of a flow ofprocesses carried out in the image processing method in accordance withEmbodiment 1. According to the processes shown in (a) of FIG. 10, it ispossible to carry out authentication of the user further quickly thanthe processes shown in FIGS. 1 and 9, although reliability is lower thanthe processes shown in FIGS. 1 and 9.

In the processes shown in (a) of FIG. 10, the image processing device 10executes authentication by the steps SB1 through SB4 after the steps S1and S2, as with the processes shown in FIG. 9. After that, theauthenticating section 15 does not carry out determination of the stepS3.

According to the processes shown in (a) of FIG. 10, authenticationitself is determined to fail when the authentication by the steps SB1through SB4 fails, even in a case where authentication would succeed ifauthentication by the steps SA1 through SA5 is carried out. Therefore,reliability of authentication is lower. Instead, the user can quicklyknow success or failure of authentication executed based on a takenpicture image.

(Forth Process of Image Processing Device 10)

(b) of FIG. 10 is a flowchart showing still another example of a flow ofprocesses carried out in the image processing method in accordance withEmbodiment 1. The processes shown in (b) of FIG. 10 are different fromthe processes shown in (a) of FIG. 10 in that the step SB2 is omitted inthe processes shown in (b) of FIG. 10. That is, according to theprocesses shown in (b) of FIG. 10, the image processing device 10removes only S-polarized light and does not remove P-polarized light.Therefore, according to the processes shown in (b) of FIG. 10, theauthenticating section 15 carries out, in the step SB4, authenticationwith use of a picture image from which a P-polarized light component ofa reflected image Ir has not been removed. Therefore, it is possible tofurther increase a processing speed as compared with the processes shownin (a) of FIG. 10, although reliability of authentication is furtherlowered.

[Experimental Example 1]

The following description will discuss, with reference to anexperimental example using a rubber ball, a reason why an image(reflected image) which is a noise included in a picture image can beremoved by the image processing device 10 through the above describedsecond process carried out by the S-polarized light calculating section12. The camera 20 used in the experiment is a camera having a CCD sensorand the number of pixels is 1900×900 (i.e., approximately 1.3 millionpixels). The integrated polarizer included in the camera 20 has fourpolarizing elements whose polarization angles are different from eachother. In the descriptions below, the polarization angles of the fourpolarizing elements are 0°, 45°, 90°, and 135°, respectively.

The polarizing element is manufactured as follows. First, a film of AlCuhaving a film thickness of 40 nm is formed, via an SiO₂ interlayer film,on a photodiode which constitutes the CCD sensor, and slits (i.e.,belt-like regions in which AlCu does not exist) each having a width of150 nm are formed by dry etching at a pitch of 300 nm. After that, afilm of SiO₂ having a film thickness of 50 nm and a film of AlCu havinga film thickness of 40 nm are formed in this order, and slits are formedin the new AlCu film such that the slits in the new AlCu film arearranged alternately with the slits of the firstly formed AlCu film.

A diameter of the rubber ball is 10 cm, and a character “G” is writtenon a surface of the rubber ball as a pattern image. In photographs fromwhich the respective picture images below are prepared, a reflectedimage overlaps with the pattern image. A distance between the rubberball and the lens of the camera 20 is 30 cm to 50 cm.

In a picture image formed by polarized light having a polarization angleof 45°, the pattern image is relatively clear. On the other hand, inpicture images formed by polarized light having a polarization angle of0° and polarized light having a polarization angle of 90°, the patternimage is obscure. In a picture image formed by polarized light having apolarization angle of 135°, the pattern image is further obscure. Thatis, among lights which have passed through the respective fourpolarizing elements in the integrated polarizer, luminance is highest inthe light which has passed through the polarizing element having thepolarization angle of 45°, and luminance is lowest in the light whichhas passed through the polarizing element having the polarization angleof 135°. Therefore, in actually measured values below, a result obtainedby calculating a difference between those two luminances of light isused.

FIG. 11 is a graph showing a reflection coefficient of S-polarized lightwith respect to an incident angle of light to the rubber ball. In FIG.11, a horizontal axis represents a ratio, to a radius of the rubberball, of a distance from a pixel which light from a center of the rubberball enters to each of pixels which light from the rubber ball enters.This ratio is a value corresponding to an incident angle of light to therubber ball. Moreover, a vertical axis represents a reflectioncoefficient of S-polarized light.

In FIG. 11, each of data points is an actually measured value.Meanwhile, a solid line constitutes a graph indicating theoreticalvalues of reflection coefficient with respect to a refractive index(n=2.2) of the rubber ball based on the formula (2-1). The actuallymeasured values obtained by the S-polarized light calculating section 12are consistent with the theoretical values. Further, based on the aboveformula (2-2) and the value of n calculated by fitting, it is possibleto obtain a luminance distribution of P-polarized light with respect tothe incident angle θ.

As above described, in Experimental Example 1, it is possible tocalculate the S-polarized light component and the P-polarized lightcomponent contained in the picture image, based on the picture imagewhich has been taken through the four polarizing elements whoseprincipal axis directions are different from each other. By subtractingthe S-polarized light component from the original picture image, it ispossible to obtain a picture image in which the pattern image is seenmore clearly than in the original picture image. Moreover, bysubtracting both the S-polarized light component and the P-polarizedlight component from the original picture image, it is possible toobtain a picture image in which the pattern image is seen further moreclearly.

Note that, in a case where the user actually takes a picture image of aneye with use of the personal digital assistant 1, the eyeball E may beoff-centered in the picture image. In a case where the eyeball E isoff-centered in the picture image, the simple relation as represented inthe formulae (1-1) and (1-2) does not hold true.

In such a case, it is possible to determine the off-centering byidentifying a position of a pixel at which the incident angle is theBrewster angle.

Specifically, dependence of luminances of pixels in each pixel unit onthe polarization angle is calculated. As above described, in the pixelunit corresponding to the Brewster angle, the P-polarized lightcomponent is zero, and accordingly a ratio of the S-polarized lightcomponent in luminance of the pixel unit becomes high. As a result,luminances of pixels included in the pixel unit largely vary dependingon the polarization angle.

Therefore, it is possible to regard a pixel unit, whose dependence onthe polarization angle is particularly large, as a pixel unitcorresponding to the Brewster angle. Further, based on a position of thepixel unit corresponding to the Brewster angle, it is possible todetermine displacement in position of the camera 20.

[Experimental Example 2]

FIG. 12 is a view showing an overview of a picture image showingexperimental results of image processing carried out by the imageprocessing device in accordance with Embodiment 1. In ExperimentalExample 2, an integrated polarizer including seven polarizing elementswhose polarization angles are different from each other is used.Moreover, in Experimental Example 2, image processing is carried out bythe steps SB1 through SB3 in the above described flowchart.

Before image processing, a picture image used in Experimental Example 2includes a reflected image Ir of outside light in the iris region, asillustrated in a part from which the arrow extends in FIG. 12. On theother hand, after the image processing, the reflected image Ir ofoutside light in the iris region has been removed, as illustrated in apart which is pointed out by the arrow in FIG. 12.

(Effect of Image Processing Device 10)

The inventor carried out experiment of authentication by a conventionaliris authentication system under various conditions, beforeaccomplishing the present invention. In the experiment, a picture imagewas taken with use of a camera of a smart phone. Positions of the smartphone with respect to an eyeball were “front” and “below”. In a casewhere the position of the smart phone is “front”, an opening degree ofeye was “large”, “medium”, or “small”. In a case where the position ofsmart phone is “below”, the opening degree of eye was “medium”.

Image taking environments were “indoor” and “outdoor”. The condition“indoor” was further divided into “no window”, “with window (shade)”,“with window (direct sunlight)”, and “darkroom”. Moreover, the condition“outdoor” was further divided into “clear sky (back light)”, “clear sky(front light)”, and “clear sky (side light)”. Note that “back light”means that the subject is irradiated with sunlight on a side opposite tothe camera. Moreover, “front light” means that the subject is irradiatedwith sunlight on a side on which the camera is located. Moreover, “sidelight” means that the subject is irradiated with sunlight on a lateralside when seen from the camera.

The following description will discuss results of authenticationexperiment which was carried out 10 times under each of the conditions.In a case where the position of the smart phone was “front”,authentication did not fail, provided that the opening degree of eye was“medium” or larger in “indoor” except for “with window (directsunlight)”. Meanwhile, “with window (direct sunlight)”, authenticationhardly succeeded. Moreover, in a case where the opening degree of eyewas “small”, authentication hardly succeeded in any of the environments.

In a case where the position of the smart phone was “below”,authentication did not fail under the condition of “indoor” except for“with window (direct sunlight)”. Further, even “with window (directsunlight)”, authentication hardly failed.

As such, in the indoor environment, authentication hardly failed unlessthe opening degree of eye was small and the subject was irradiated withdirect sunlight through the window.

However, in a case where the environment was “outdoor” and the positionof the smart phone was “front”, authentication did not succeed at all.In a case where the position of the smart phone was “below”,authentication could be carried out without problem under “clear sky(side light)”, but the number of succeeded authentication under each of“clear sky (back light)” and “clear sky (front light)” was not more thana half of the total.

As above described, the authentication carried out with the conventionaliris authentication system has a problem that authentication hardlysucceeds in the outdoor environment. This is because, as earlydescribed, outside light or the like is reflected on the eyeball.

Patent Literature 1 does not disclose that the reflection component isseparated by obtaining a luminance distribution of a specular reflectioncomponent and, furthermore, that iris authentication is carried out byremoving an image of another object reflected on an eyeball which is asubject. Moreover, Patent Literature 1 does not disclose that a process,which is different from the process of removing the image of the object,is carried out for iris authentication.

According to the image processing device in accordance with Embodiment1, the S-polarized light calculating section 12 obtains a luminancedistribution of an S-polarized light component which luminancedistribution depends on the incident angle θ, and this makes it possibleto calculate and remove an S-polarized light component that is caused bya reflected image Ir included in the picture image of the eyeball.Moreover, according to the image processing device of Embodiment 1, itis possible to calculate and remove, based on the S-polarized lightcomponent, a P-polarized light component that is caused by the reflectedimage Ir.

That is, according to the image processing device in accordance withEmbodiment 1, it is possible to remove a reflected image caused byoutside light on the eyeball. By carrying out iris authentication basedon the picture image from which the reflected image caused by outsidelight has been removed, it is possible to carry out authentication withhigh accuracy, regardless of environments.

Moreover, in the image processing method of Patent Literature 1, it isnecessary to carry out the processes (1) through (4) in order toseparate a diffuse reflection component from a specular reflectioncomponent in the process (5). Therefore, in the image processing methodof Patent Literature 1, an algorithm for separating those two reflectioncomponents becomes complicated, and consequently an arithmeticaloperation speed in the image processing may decrease. In the imageprocessing device 10 in accordance with Embodiment 1, as abovedescribed, it is possible to separate the two reflection components fromeach other, without carrying out the processes (1) through (4). It istherefore possible to improve an arithmetical operation speed in theimage processing, as compared with the image processing method of PatentLiterature 1.

[Embodiment 2]

The following description will discuss Embodiment 2 of the presentinvention, with reference to FIG. 4 and FIG. 5. Note that, forconvenience of explanation, the same reference numerals are given toconstituent members which have functions identical with those describedin Embodiment 1, and descriptions regarding such constituent members areomitted.

The camera 20 in accordance with Embodiment 1 has nine polarizingelements, i.e., the polarizing elements 21 a through 21 i whichconstitute the integrated polarizer 21. Meanwhile, as illustrated in (a)of FIG. 5, the camera 20 in accordance with Embodiment 2 includes theintegrated polarizer 21A which has four polarizing elements, i.e., thepolarizing elements 21 j through 21 m corresponding to respective fourpixels that are adjacent to each other. The polarizing elements 21 j, 21k, 21 l, and 21 m respectively have polarization angles of 135°, 90°,45°, and 0° in the respective pixels. As such, in Embodiment 2, theintegrated polarizer 21A is used instead of the integrated polarizer 21.In this case also, as with Embodiment 1, it is possible to calculate aluminance value of S-polarized light in the steps SA1 and SB1 shown inFIGS. 1, 9, and 10. That is, as with Embodiment 1, it is possible toremove a specular reflection component from reflected light Lr.

However, the number of the polarizing elements in the integratedpolarizer 21A of Embodiment 2 is four (i.e., the polarizing elements 21j through 21 m), which is smaller than that in Embodiment 1.

In general, in characteristics between a polarization angle of anintegrated polarizer and a received-light intensity of reflected lightLr received by a pixel, the received-light intensity forms asubstantially sine function or a substantially cosine function. InEmbodiment 1, in one (1) pixel unit, received-light intensities ofreflected light Lr in the nine polarizing elements, i.e., the polarizingelements 21 a through 21 i are obtained. Therefore, as illustrated inFIG. 7, it is possible to fit a waveform exhibited by received-lightintensities in the characteristics into the substantially sine functionor the substantially cosine function. Therefore, for example, a maximumvalue (maximum luminance value) and minimum value (minimum luminancevalue) of received-light intensities in one (1) pixel unit used in thestep SA1 shown in FIG. 1 are highly likely to substantially conform torespective of a maximum luminance value and a minimum luminance value inthe pixel unit.

Meanwhile, in Embodiment 2, fitting into the substantially sine functionor the substantially cosine function is carried out by interpolation,and there is a possibility that accuracy in fitting is lowered and amaximum value and a minimum value of the substantially sine function orthe substantially cosine function which has been subjected to fittingbecome different from the maximum luminance value and the minimumluminance value, as compared with Embodiment 1.

However, in Embodiment 2, one (1) pixel unit which corresponds to theintegrated polarizer 21A is made up of four pixels. That is, the numberof integrated polarizers 21A is larger than that in Embodiment 1.Therefore, for example, it is possible to obtain a pixel correspondingto the Brewster angle in a more segmentalized manner than Embodiment 1.That is, it is possible to identify the pixel more accurately.

As such, in Embodiment 2, accuracy in fitting may be lowered but it ispossible to identify the pixel corresponding to the Brewster angle moreaccurately, as compared with Embodiment 1. Therefore, it is possible todetermine, by analyzing accuracy in iris authentication, which one ofthe integrated polarizer 21 of Embodiment 1 and the integrated polarizer21A of Embodiment 2 is to be used.

Note that, by carrying out the analysis, it is possible to employ anintegrated polarizer 21B (including two polarizing elements 21 n havinga polarization angle of 90° and two polarizing elements 210 having apolarization angle of 0°) illustrated in (b) of FIG. 5, instead of theintegrated polarizer 21A. Moreover, the polarization angle, and thenumber of polarizing elements corresponding to one (1) pixel unit can bechanged as appropriate, in accordance with the analysis.

[Embodiment 3]

The following description will discuss Embodiment 3 of the presentinvention, with reference to FIGS. 13 and 14. In authentication carriedout with use of an iris picture image, there may be a case whereauthentication is carried out by impersonating the user by using animitation of the eyeball E. Therefore, it is necessary to take intoconsideration prevention of such impersonation.

FIG. 13 is a block diagram illustrating a configuration of a personaldigital assistant 1A including an image processing device 10A ofEmbodiment 3. As illustrated in FIG. 13, the image processing device 10Ais different from the image processing device 10 in that theauthenticating section 15 includes an impersonation determining section15 a in the image processing device 10A.

In general, a refractive index of a cornea of an eye of a living body isn=1.376. On the other hand, in a case of an imitation of an eye, arefractive index is different from that of an eye of a living bodybecause a component and a material of the imitation are different fromthose of the eye of the living body. The impersonation determiningsection 15 a determines whether or not an eye is a living body or is animitation, based on that difference.

Specifically, the impersonation determining section 15 a refers to avalue of n which has been calculated by the S-polarized lightcalculating section 12 in the step SB1. In a case where the value of nis equal or near to 1.376 (i.e., a value falling within a predeterminedrange, e.g., within ±5%), the impersonation determining section 15 adetermines that a taken picture image is of an eyeball E of a livingbody. On the other hand, in a case where the calculated value of n isaway from 1.376 (i.e., out of the predetermined range, e.g., a valuedifferent by more than ±5%), the impersonation determining section 15 adetermines that a taken picture image is of an imitation of an eyeballE.

FIG. 14 is a flowchart showing processes in the image processing device10A. As shown in FIG. 14, processes in the image processing device 10Aare different from those shown in FIG. 1 only in that a step SB5 isexecuted between the step SB3 and the step SB4.

In the step SB5, the impersonation determining section 15 a determineswhether or not a value of n calculated by the S-polarized lightcalculating section 12 in the step SB1 falls within a predeterminedrange. In a case where the value of n falls within the predeterminedrange (Y in SB5), the impersonation determining section 15 a determinesthat a taken picture image is of an eyeball E of a living body, andcarries out the process of SB4. On the other hand, in a case where thevalue of n does not fall within the predetermined range (N in SB5), theimpersonation determining section 15 a determines that a taken pictureimage is of an imitation of an eyeball E, and ends authenticationwithout carrying out the process of SB4.

As above described, according to the image processing method inaccordance with Embodiment 3, luminance distribution of an S-polarizedlight component depending on the incident angle θ is obtained by theS-polarized light calculating section 12, and the luminance distributionof S-polarized light can be used to determine whether the subject is aneyeball or an imitation.

Note that the impersonation determining section 15 a can regard, as theBrewster angle, an angle at which a luminance value becomes a minimumvalue in a luminance distribution of a P-polarized light componentcalculated by the P-polarized light calculating section 13 in the stepSB2. Then, the impersonation determining section 15 a can determinewhether the subject is an eyeball E of a living body or an imitation ofan eyeball E depending on whether or not the angle is near to 53.1°.That is, the impersonation determining section 15 a can carry out thedetermination based on whether or not the angle falls within apredetermined range within which the subject is assumed to be an eyeballE of a living body and which includes the Brewster angle of the eyeballE of the living body.

Moreover, in the processes of FIGS. 9 and 10 also, the determination canbe carried out by executing the step SB5 between the step SB3 and thestep SB4.

[Modification Example]

The following description will discuss another method for preventingimpersonation by an imitation.

In a case where removal of a reflected image is carried out with respectto an imitation whose refractive index is different from that of aneyeball E of a living body while assuming that the Brewster angle is53.1°, a reflected image due to outside light or the like to theimitation is not properly removed. Therefore, in a case where areflected image due to outside light or the like exists, it is highlypossible without calculating a refractive index that authenticationfails because an iris pattern changes due to the reflected image.

However, in a case where authentication is carried out in an environment(e.g., a darkroom) in which a reflected image does not occur, there is apossibility that an iris pattern does not change and authenticationsucceeds.

In view of this, the image processing device can include a light sourcefor purposely causing a reflected image on an eyeball E, as a method forpreventing impersonation by an imitation. A reflected image thus caused(i.e., an image formed by the light source) is properly removed by theimage processing device if the subject is an eyeball E of a living body,and therefore authentication is not influenced. On the other hand, in acase of an imitation of an eyeball E, the image processing device cannotremove the reflected image, and therefore authentication fails. Notethat an intensity of light emitted by the light source is not limited,provided that a reflected image is caused by the light. Therefore, forexample, the intensity does not need to be a high intensity that causesan adverse influence on a human body.

[Embodiment 4]

The following description will discuss Embodiment 4 of the presentinvention with reference to FIGS. 15 through 17.

FIG. 15 is a block diagram illustrating a configuration of an electronicinformation apparatus 2 which includes an image processing device 10B inaccordance with Embodiment 4. As illustrated in FIG. 15, the electronicinformation apparatus 2 includes an image processing device 10B, acamera 20, and a storage section 90. Moreover, the image processingdevice 10B includes a luminance value information obtaining section 16,a minimum luminance value selecting section 17 (image generatingsection), an iris detecting section 11, and an authenticating section15.

The luminance value information obtaining section 16 obtains effectiveradiance values of a subject by an image pickup device included in thecamera 20. Each of the effective radiance values is an effectiveradiance value of the subject in the image pickup device. Specifically,the luminance value information obtaining section 16 obtains, aseffective radiance values of respective pixels of the image pickupdevice included in the camera 20, intensities of light which has beenreflected by the subject and then received in the pixels. Note that, inEmbodiment 4 and subsequent embodiments, the “subject” can be either aneyeball E of one eye or eyeballs E of both eyes or can include aneyeball E and a surrounding object(s).

The minimum luminance value selecting section 17 generates a pictureimage including an image of an iris, with use of effective radiancevalues obtained from the iris of the subject. Specifically, the minimumluminance value selecting section 17 obtains, for each of pixel unitscorresponding to the iris of the subject, a luminance value by at leastpartially removing a regular reflection light component on a surface ofthe iris from effective radiance values of a plurality of pixelsincluded in that pixel unit, and generates a picture image as a set ofunit regions having obtained luminance values. Moreover, the sameapplies to the above described diffusion light calculating section 14,and a minimum luminance value estimating section 17A and a diffusivereflection light component calculating section 18 which will bedescribed later.

Specifically, the minimum luminance value selecting section 17 ofEmbodiment 4 determines, as a luminance value of a unit regioncorresponding to a pixel unit, a minimum value of effective radiancevalues of a plurality of pixels included in the pixel unit. Pixel unitscorresponding to the iris of the subject are identified by the irisdetecting section 11 in advance. The unit regions are in the pictureimage of the subject and correspond to the respective pixel units in thecamera 20.

In Embodiment 4 and the subsequent embodiments, a component of reflectedlight forming a reflected image is referred to as “regular reflectionlight component”. The regular reflection light component includes acomponent which depends on a principal axis direction of a polarizingelement that is provided so as to correspond to a pixel. A minimum valueof effective radiance values in a pixel unit can be considered as aneffective radiance value from which a component depending on theprincipal axis direction of the polarizing element has been mostlyremoved in the pixel unit. Therefore, the minimum luminance valueselecting section determines the minimum value of effective radiancevalues in the pixel unit as a luminance value of the unit region, andthus generates a picture image as a set of unit regions having luminancevalues obtained by at least partially removing a regular reflectionlight component on the surface of the cornea of the subject.

The storage section 90 is a storage device for storing informationnecessary for the image processing device 10B to execute a process. Notethat it is possible to employ a configuration in which the electronicinformation apparatus does not include the storage section 90 and cancommunicate with a storage device that is provided outside theelectronic information apparatus 2.

FIG. 16 is a flowchart showing processes in the image processing device10B. In the image processing device 10B, first, the luminance valueinformation obtaining section 16 obtains, from the camera 20, effectiveradiance values of the subject including the iris (SC1, luminance valueinformation obtaining step). Next, the iris detecting section 11 detectsan iris region based on the effective radiance values (SC11). Theminimum luminance value selecting section 17 determines the iris region,which has been detected by the iris detecting section 11, as a rangewithin which an image processing is to be executed (SC12). The minimumluminance value selecting section 17 determines a minimum value, amongeffective radiance values of a plurality of pixels in each of pixelunits in the iris region, as a luminance value of a unit regioncorresponding to that pixel unit, and thus at least partially removes aregular reflection light component on the surface of the cornea (SC2,image generating step). Further, the minimum luminance value selectingsection 17 generates a picture image as a set of unit regions having theluminance values determined in the step SC2 (SC3, image generatingstep).

The iris detecting section 11 executes a process of detecting an irisregion, with respect to the picture image which has been generated bythe minimum luminance value selecting section 17 (SC4). The irisdetecting section 11 can use a detection result of the iris region inthe step SC11 directly as a detection result in the step SC4. Theauthenticating section 15 carries out authentication of a user with useof the iris region which has been detected (SC5). After that, theauthenticating section 15 determines whether or not the user has beenauthenticated (SC6). In a case where the user has been authenticated (Yin SC6), the image processing device 10B ends the process. In a casewhere the user has not been authenticated (N in SC6), the iris detectingsection 11 detects an iris region again.

In the case of N in the step SC6, there is a possibility that the irisdetecting section 11 has not properly detected an iris region of theuser. In view of this, in the step SC4 after the step SC6 is N, it ispreferable that an iris region to be detected by the iris detectingsection 11 is different from the iris region which has been previouslydetected in the step SC4. A concrete example of a method for making irisregions different can be increasing or decreasing a diameter of the irisregion. As such, it is preferable to apply feedback to the process ofdetecting an iris region by the iris detecting section 11 such that ahumming distance between a registered code of an eyeball of the user anda code of an eyeball E in a taken image becomes shortest.

FIG. 17 is a graph showing an example of effective radiance values ofpixels in a pixel unit. In the graph shown in FIG. 17, a horizontal axisrepresents an angle of a principal axis of a polarizing element whichcorresponds to a pixel included in a pixel unit of the camera 20, and avertical axis represents an effective radiance value of the pixel in thepixel unit. In the example shown in FIG. 17, the pixel unit includesnine pixels which correspond to respective nine polarizing elementswhose principal axis directions are different from each other by 20°starting from 10°.

In the example shown in FIG. 17, a minimum value of the effectiveradiance value is of a pixel in the pixel unit which pixel correspondsto a polarizing element whose principal axis direction is 90°, and amaximum value of the effective radiance value is of a pixel whichcorresponds to a polarizing element whose principal axis direction is170°. In this case, the minimum luminance value selecting section 17selects the effective radiance value of the pixel, which corresponds tothe polarizing element whose principal axis direction is 90°, as aluminance value of a unit region corresponding to the pixel unit.

In the image processing device 10B, a minimum value of effectiveradiance values of pixels included in each of pixel units is determinedas a luminance value of a unit region corresponding to that pixel unit.From this, it is possible to generate a picture image by simplydetermining a luminance value of a unit region without needing acomplicated computation process.

Note that, in the image processing device 10B, the steps SC11 and SC12can be omitted from the steps in the above described flowchart. In sucha case, the minimum luminance value selecting section 17 may determineluminance values of respective unit regions corresponding to all pixelunits. Moreover, in an image processing device 10C of Embodiment 5 whichwill be described later also, the steps SC11 and SC12 can be omitted.Note, however, that, in image processing devices 10D and 10E inEmbodiments 6 through 8 which will be described later, the steps SC11and SC12 cannot be omitted. Moreover, from the viewpoint of processingspeed, it is preferable to determine luminance values of only unitregions corresponding to pixel units that correspond to the iris region.Therefore, it is preferable to execute the steps SC11 and SC12 in theimage processing devices 10B and 10C.

Each of the image processing devices 10B and 10C in Embodiments 4 and 5can carry out a process of identifying a region (e.g., a region largerthan the iris by a predetermined range, a region corresponding to theentire eyeball E, or the like) that is different from the iris regionbefore the step SC2. In such a case, the minimum luminance valueselecting section 17 or a minimum luminance value estimating section 17A(see Embodiment 5) may execute an image processing with respect to theidentified region. That is, each of the image processing devices 10B and10C may determine luminance values in unit regions corresponding topixel units corresponding to at least the iris (i.e., at least a part)of the subject based on effective radiance values of a plurality ofpixels included in the pixel units.

[Embodiment 5]

The following description will discuss Embodiment 5 of the presentinvention with reference to FIGS. 18 through 20.

FIG. 18 is a block diagram illustrating a configuration of an electronicinformation apparatus 2 including an image processing device 10C inaccordance with Embodiment 5. As illustrated in FIG. 18, the imageprocessing device 10C is substantially identical with the imageprocessing device 10B except that the image processing device 10Cincludes a minimum luminance value estimating section 17A (imagegenerating section) instead of the minimum luminance value selectingsection 17.

The minimum luminance value estimating section 17A estimates a minimumvalue of possible effective radiance values in each of pixel units basedon effective radiance values of a plurality of pixels included in thatpixel unit. A concrete example of a method for estimating the minimumvalue will be described later.

Further, the minimum luminance value estimating section 17A determines aluminance value of a unit region based on the estimated minimum value ofeffective radiance value. Here, the minimum luminance value estimatingsection 17A can set the estimated minimum value to be a luminance valueof the unit region or can set a value, which is obtained by carrying outa predetermined calculation with respect to the estimated luminancevalue, to be a luminance value of the unit region. For example, theminimum luminance value estimating section 17A can set a value, whichhas been obtained by carrying out addition, subtraction, multiplication,or division with a predetermined value with respect to the estimatedminimum value of effective radiance value, to be a luminance value ofthe unit region.

FIG. 19 is a flowchart showing processes in the image processing device10C. The processes in the image processing device 10C are substantiallyidentical with those in the image processing device 10B, except that astep SC7 (image generating step) is executed instead of the step SC2. Inthe step SC7, the minimum luminance value estimating section 17Aestimates minimum values of possible effective radiance values inrespective pixel units corresponding to the eyeball E.

FIG. 20 is a graph showing an example of effective radiance values ofpixels in a pixel unit. In the graph shown in FIG. 20, a horizontal axisrepresents an angle of a principal axis of a polarizing elementcorresponding to each of pixels included in a pixel unit, and a verticalaxis represents an effective radiance value. In the example shown inFIG. 20, the pixel unit includes four pixels corresponding to respectivefour polarizing elements whose principal axis directions are differentfrom each other by 45° starting from 10°.

In the example shown in FIG. 20, effective radiance values obtained inthe pixels included in the pixel unit become smallest in a pixelcorresponding to a polarizing element whose principal axis direction is100°. However, in a case where a relation between the principal axisdirection and the effective radiance value corresponds to, for example,a function shown by a dashed line in FIG. 20, a smaller effectiveradiance value may be obtained in a pixel included in the pixel unit,depending on a principal axis direction of a polarizing elementcorresponding to the pixel. In a case where a minimum value of actualeffective radiance values of the pixels in the pixel unit is determinedas a luminance value of the unit region under the above condition, theluminance value still contains a regular reflection light componentdepending on the principal axis directions of the polarizing elements.In the image processing device 10C, the minimum luminance valueestimating section 17A estimates a minimum value of possible effectiveradiance values in the pixel unit, and it is possible to generate apicture image containing less regular reflection light component, ascompared with a picture image generated in the image processing device10B.

In the example described below, the minimum luminance value estimatingsection 17A applies, to effective radiance values of a plurality ofpixels included in each of pixel units, a trigonometric function inwhich an angle of a principal axis direction of a polarizing elementcorresponding to each of the pixels is a variable, and thereby estimatesthe minimum value. As such, the minimum luminance value estimatingsection 17A can estimate a minimum value of possible luminance values inthe unit region, based on the minimum value of the trigonometricfunction.

(First Estimating Method)

The following description will discuss an example of a method forestimating a minimum luminance value in a unit region by the minimumluminance value estimating section 17A. It is assumed that, in a casewhere an effective radiance value of each of pixels in a certain pixelunit is Y, Y can be expressed by a formula (5-1) below.Y=Asin(2x+B)+C  (5-1)

Here, each of A and B is a constant unique to the pixel unit. x is anangle of a principal axis of a polarizing element corresponding to eachof the pixels. In a case where principal axis directions of polarizingelements corresponding to respective pixels in the pixel unit aredifferent from each other by a constant angle, C conforms to an averageE(Y) of effective radiance values of the respective pixels in the pixelunit.

In a case where a mean square of Y is E(Y²), E(Y²) is expressed by aformula (5-2) below.E(Y ²)=A ² ×E(sin²(2x+B))+2A×E(C)×E(sin(2x+B))+(E(C ²))  (5-2)

Here, from Wallis formulas, formulae (5-3) and (5-4) below hold true.E(sin²(2x+B))=1/2  (5-3)E(sin(2x+B))=1/π  (5-4)

Therefore, the formula (5-2) can be modified into a formula (5-5) below.E(Y ²)=A ²/2+2A×E(C)/π+(E(C ²))  (5-5)

E(Y²) is equal to a mean square of the effective radiance values of therespective pixels included in the pixel unit, and therefore it ispossible to calculate a value of A by approximating E(C) to a minimumvalue of effective radiance values of the respective pixels,approximating E(C²) to a square value of the minimum value of effectiveradiance values of the respective pixels, and solving the formula (5-5)for A.

Based on the value of A calculated by solving the formula (5-5) and onthe formula (5-1), it is possible to obtain a minimum value Ymin ofpossible values of Y by a formula (5-6) below.Ymin=E(Y)−A  (5-6)

As such, the minimum luminance value estimating section 17A can estimatea minimum value of possible effective radiance values of the pixelsincluded in the pixel unit, only by the calculations by the formulae(5-5) and (5-6).

FIG. 21 is a graph showing parameters included in the formula (5-1). Inthe graph shown in FIG. 21, a horizontal axis represents an angle of aprincipal axis of each of polarizing elements in a pixel unit of thecamera 20, and a vertical axis represents an effective radiance value ofa pixel in the pixel unit. In the example shown in FIG. 21, the pixelunit includes four pixels corresponding to respective four polarizingelements whose principal axis directions are different from each otherby 45° starting from 10°.

As shown in FIG. 21, the value of C in the formula (5-1) is a medianvalue of effective radiance values of the pixels in the pixel unit, andis substantially equal to an average value of the effective radiancevalues of the pixels in the pixel unit. Moreover, a value of A in theformula (5-1) is a difference between the average value and a possibleminimum luminance value of the pixels in the unit region. Moreover, avalue of B in the formula (5-1) is a value for determining a position ofa graph in the horizontal-axis direction. That is, in a case where thevalue of B varies, the entire graph shifts in parallel to the horizontalaxis. The value of B does not influence Ymin, and therefore the value ofB is not calculated in the first estimating method.

(Second Estimating Method)

The following description will discuss a second method for estimating aminimum luminance value in a unit region by the minimum luminance valueestimating section 17A. In the second estimating method also, it isassumed that an effective radiance value Y of each of pixels included ina certain pixel unit can be expressed by the formula (5-1).

In the second estimating method, a look-up table showing correspondencebetween values of x and values of sin 2x is stored in the storagesection 90 in advance. The minimum luminance value estimating section17A sets an initial value of A in the formula (5-1) to an average of (i)and (ii) below, and prepares a table of Y corresponding to values of xwith reference to the look-up table.

(i) A difference between a maximum value and an average value ofeffective radiance values of respective pixels in a pixel unit

(ii) A difference between an average value and a minimum value ofeffective radiance values of respective pixels in a pixel unit

The correspondence between x and Y in the prepared table can be shiftedby changing the value of B in the formula (5-1).

Next, the minimum luminance value estimating section 17A compares theprepared table with the effective radiance values of the respectivepixels included in the pixel unit. The minimum luminance valueestimating section 17A changes the value of B such that a total ofdifferences between (i) effective radiance values of respective pixelsincluded in the pixel unit and (ii) values in the table at angles ofpolarizing elements corresponding to the respective pixels becomessmallest. In a case where the total of differences between the values inthe table and the effective radiance values is smallest and an averageof ratios of (i) differences between values in the table and theeffective radiance values of the respective pixels to (ii) the effectiveradiance values of the respective pixels is equal to or less than apredetermined ratio, the minimum luminance value estimating section 17Aestimates that the value of A is appropriate. In this case, the minimumluminance value estimating section 17A estimates a minimum value by theformula (5-6). The predetermined ratio is preferably 10%, morepreferably 5%, and further preferably 1%.

In a case where the difference between the values of the table and theeffective radiance values is not equal to or less than the predeterminedratio, the minimum luminance value estimating section 17A changes thevalue of A in the formula (5-1) from the initial value, and carries outsimilar comparison.

(a) of FIG. 22 is a view showing an example of the look-up table. (b) ofFIG. 22 is a view showing an example of the table prepared by theminimum luminance value estimating section 17A. (c) of FIG. 22 is a viewshowing an example of signal intensities indicative of effectiveradiance values. (d) of FIG. 22 is a graph showing an example of thesignal intensities shown in (c) of FIG. 22 and an example of signalintensities indicative of effective radiance values estimated by theminimum luminance value estimating section 17A.

In the graph shown in (d) of FIG. 22, a horizontal axis represents anangle of a principal axis of each of polarizing elements in a pixel unitof the camera 20, and a vertical axis represents a signal intensityindicative of an effective radiance value of a pixel in the pixel unit.In the example shown in (d) of FIG. 22, the pixel unit includes fourpixels corresponding to respective four polarizing elements whoseprincipal axis directions are different from each other by 45° startingfrom 0°. The example of (d) of FIG. 22 shows a case where light entersan interface between air (refractive index n1=1.0) and a cornea(refractive index n2=1.376) at an incident angle of 15°.

The storage section 90 stores the look-up table as shown in (a) of FIG.22 in advance. The minimum luminance value estimating section 17A setsthe value of A in the formula (5-1) as above described, and prepares atable of Y as shown in (b) of FIG. 22. Further, the minimum luminancevalue estimating section 17A extracts, from the table of Y, values whichare nearest to respective effective radiance values of the pixelsincluded in the pixel unit as shown in (c) of FIG. 22. The extractedvalues and principal axis directions of polarizers corresponding to thevalues are indicated by “*” in (d) of FIG. 22. Meanwhile, actualeffective radiance values of respective pixels included in the pixelunit and principal axis directions of polarizers corresponding to therespective pixels are indicated by “∘” in (d) of FIG. 22.

In (d) of FIG. 22, positions of “*” in the vicinity of 0° and in thevicinity of 90° well conform to respective positions of “∘”, whereaspositions of “*” in the vicinity of 65° and in the vicinity of 155° arefar apart from positions of “∘” at 45° and 135°. Therefore, the abovedescribed ratio is greater than 10%. In this case, the minimum luminancevalue estimating section 17A (i) prepares a table of Y again by changingthe value of A in the formula (5-1), (ii) extracts, from the table of Y,values which are nearest to respective actual effective radiance values,and (iii) carries out similar comparison.

(Another Estimating Method)

The minimum luminance value estimating section 17A can estimate aminimum value of possible effective radiance values of pixels in a pixelunit by an estimating method other than the above described first andsecond estimating methods. For example, the minimum luminance valueestimating section 17A can estimate the minimum value by applying apolynomial expression, in which an angle of a principal axis directionof a polarizing element corresponding to each of a plurality of pixelsin each of pixel units is a variable, to effective radiance values ofthe plurality of pixels. In this case, the polynomial expression can be,for example, a polynomial expression obtained by subjecting atrigonometric function to Taylor expansion. According to Taylorexpansion, sin x and cos x can be expressed as in respective formulae(5-7) and (5-8) below.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{{\sin\mspace{11mu} x} = {\sum\limits_{n = 0}^{\infty}\;{\frac{\left( {- 1} \right)^{n}}{\left( {{2\; n} + 1} \right)!}x^{{2\; n} + 1}}}} & \left( {5\text{-}7} \right) \\\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack & \; \\{{\cos\mspace{11mu} x} = {\sum\limits_{n = 0}^{\infty}\;{\frac{\left( {- 1} \right)^{n}}{\left( {2\; n} \right)!}x^{2\; n}}}} & \left( {5\text{-}8} \right)\end{matrix}$

In the formulae (5-7) and (5-8), x is a principal axis direction of apolarizing element, and n is an arbitrary integer. n is preferably 3 ormore.

Moreover, for example, it is possible that the minimum luminance valueestimating section 17A estimates a function indicative of effectiveradiance values of pixels in a pixel unit by an arbitrary interpolatingmethod, and estimates the minimum value by use of the function. Examplesof the interpolating method encompass spline interpolation, polynomialinterpolation, trigonometric function interpolation, cubicinterpolation, and the like.

[Embodiment 6]

The following description will discuss Embodiment 6 of the presentinvention with reference to FIGS. 23 through 25.

FIG. 23 is a block diagram illustrating a configuration of an electronicinformation apparatus 2 including an image processing device 10D inaccordance with Embodiment 6. As illustrated in FIG. 23, the imageprocessing device 10D is different from the image processing device 10Bin that the image processing device 10D includes a diffusive reflectionlight component calculating section 18 (image generating section)instead of the minimum luminance value selecting section 17.

The diffusive reflection light component calculating section 18identifies a maximum value Ioutmax and a minimum value Ioutmin amongeffective radiance values in each of pixel units. Further, the diffusivereflection light component calculating section 18 calculates a diffusivereflection light component Ik in each of pixel units by a formula (6-1)below.Ik=(Ioutmin−a×Ioutmax)/(1−a)  (6-1)

Here, Ioutmin and Ioutmax are respectively a minimum value and a maximumvalue of effective radiance values of pixels included in a pixel unit.Ioutmin and Ioutmax can be values estimated in a manner similar to thatof Embodiment 5, or can be actual effective radiance values of pixels.

Moreover, a in the formula (6-1) represents a ratio of a reflectance ofa P-polarized light component to a reflectance of an S-polarized lightcomponent. As above described, a depends on an incident angle of lightto the eyeball E. Based on a distance R from the lens of the camera 20to a center of the eyeball E and a radius r of the eyeball E, it ispossible to calculate the incident angle at each point in a pictureimage of the eyeball E. In Embodiment 6, a distance R measured while theeyeball E is in focus is stored in the storage section 90 in advance,and the diffusive reflection light component calculating section 18calculates the incident angle and the value of a with use of thedistance R.

The distance R is specifically obtained as follows. In a case where thecamera 20 is a camera having a fixed focus lens, the camera 20 isdesigned such that a face of a user is focused when the user sees thescreen while holding the electronic information apparatus 2 in theuser's hand. With the configuration, in a case where the user takes apicture image of a region including the user's own eye such that apicture image of an eyeball E is included in a predetermined range ofthe picture image taken with the camera 20, a distance R between thelens of the camera 20 and the eyeball E becomes substantially equal to afocal distance of the lens of the camera 20. In a case where the camera20 has an autofocus function, a distance R can be calculated with theBessel method by obtaining two kinds of distances between lenses forimage formation of the eyeball E. In a case where the camera 20 is acamera which includes two image pickup devices and can simultaneouslytake a plurality of picture images from different positions, it ispossible to calculate a distance R by triangulation based on the pictureimages taken by the respective image pickup devices.

Here, in a case where an incident angle is small, a difference inreflectance between an S-polarized light component and a P-polarizedlight component is small, i.e., approximately several tens of percent.Therefore, a value of a cannot be accurately calculated due to influenceof unevenness between pixels or shots, and sometimes a result ofcalculation with the formula (6-1) becomes as follows: Ik≤0 (i.e., adiffusive reflection light component disappears).

In this case, calculation by the formula (6-1) can be executed bychanging the value of a into a value of a which corresponds to an anglegreater than an actual incident angle. For example, calculation by theformula (6-1) can be sequentially executed again for values of acorresponding to angles which sequentially become greater than theactual incident angle by a predetermined angle (e.g., 10°). Note,however, that, in this case, accuracy in removing a reflected imagedecreases. Such recalculation by changing the value of a may be executedin a case where, for example, authentication by the authenticatingsection 15 has failed. Alternatively, in such recalculation, it ispossible to change the value of a into a value corresponding to a radiussmaller than an actual radius r or a distance shorter than an actualdistance R, instead of changing the value of a into a valuecorresponding to an angle greater than an actual incident angle.

Alternatively, the incident angle and the value of a can be calculatedbased on a distance R which is assumed in advance. For example, in acase where the electronic information apparatus 2 is a smart phone, itis assumed that the user carries out authentication in a state where theuser is holding the electronic information apparatus 2 in the user'shand. In such a case, the distance R would be approximately 20 cm to 30cm. Therefore, it is possible to calculate the incident angle based onthe expected distance R, without actually measuring a distance R with adistance measuring device or the like.

In this case, it is possible to prepare a plurality of distances R. Thatis, authentication can be executed by calculating the incident angle andthe value a and further calculating the diffusive reflection lightcomponent Ik for each of the plurality of distances R. In the aboveexample, the image processing device 10D can execute authentication foreach of cases where the distance R is 20 cm and where the distance R is30 cm, and may further execute authentication for a case where thedistance R is 25 cm.

The following description will discuss the formula (6-1). Each ofeffective radiance values of pixels included in a pixel unit includes adiffusive reflection light component and a regular reflection lightcomponent. In Embodiment 6, the regular reflection light componentincludes a first regular reflection light component and a second regularreflection light component. Moreover, in Embodiment 6, each of the firstregular reflection light component and the second regular reflectionlight component has a component which depends on a principal axisdirection of a polarizing element. In this case, a first regularreflection light component Is and a second regular reflection lightcomponent Ip are respectively expressed by formulae (6-2) and (6-3)below.Is=Is0+Is0×cos 2x  (6-2)Ip=Ip0+Ip0×cos 2(x−π/2)  (6-3)

Here, Is0 and Ip0 are respectively components of the first regularreflection light component and the second regular reflection lightcomponent which components do not depend on a principal axis directionof a polarizing element. Moreover, a phase of the second regularreflection light component is delayed from a phase of the first regularreflection light component by 90°, and therefore π/2 is subtracted fromx in the formula (6-3). Note that, in the formulae (6-2) and (6-3), thecosine function can be substituted by a sine function.

An effective radiance value Iout of each of pixels included in a pixelunit is expressed by a formula (6-4) below.Iout=Ik+Is+Ip=Ik+Is0+Ip0+(Is0−Ip0)cos 2x  (6-4)

Iout becomes a maximum value when cos 2x=1, and becomes a minimum valuewhen cos 2x=−1. Therefore, Ioutmax and Ioutmin in the formula (6-1) arerespectively expressed by formulae (6-5) and (6-6) below.Ioutmax=Ik+2×Is0  (6-5)Ioutmin=Ik+2×Ip0  (6-6)

Moreover, from Fresnel's law, a relation between Is0 and Ip0 isexpressed by a formula (6-7) below.Ip0=a×Is0  (6-7)

Therefore, by substituting the formula (6-7) for the formula (6-6), aformula (6-8) below is obtained.Ioutmin=Ik+2×a×Is0  (6-8)

By solving simultaneous equations including the formulae (6-5) and (6-8)for Ik, the formula (6-1) is obtained.

FIG. 24 is a flowchart showing processes in the image processing device10D. The processes in the image processing device 10D are substantiallyidentical with the processes in the image processing device 10B, exceptthat a step SC8 (image generating step) is executed instead of the stepSC2. In the step SC8, the diffusive reflection light componentcalculating section 18 calculates a diffusive reflection light componentIk for each of pixel units by the formula (6-1), and determines thecalculated value as a luminance value of a unit region corresponding tothat pixel unit.

The above described recalculation by changing the value of a can beexecuted, for example, as follows. That is, a threshold (hereinafter,referred to as “branch threshold”), which is greater than a threshold(hereinafter, referred to as “authentication threshold”) used inauthentication, is set to a humming distance between (i) a code of apicture image of the eyeball from which picture image a reflected imagehas been removed and (ii) a registered code. In the step SC6 in FIG. 24,in a case where the humming distance is equal to or greater than thebranch threshold, the diffusive reflection light component calculatingsection 18 executes the process of the step SC8 again. In this case,recalculation by changing the value of a which is described above isexecuted. On the other hand, in a case where the humming distance isequal to or greater than the authentication threshold and less than thebranch threshold in the step SC6, the iris detecting section 11 executesthe process of the step SC4 again. Note that, in a case where thehumming distance is less than the authentication threshold,authentication succeeds.

As above described, the diffusive reflection light component calculatingsection 18 of Embodiment 6 identifies a maximum value and a minimumvalue of effective radiance values in each of pixel units based oneffective radiance values of a plurality of pixels included in thatpixel unit. The diffusive reflection light component calculating section18 calculates a diffusive reflection light component Ik in that pixelunit, with use of the identified maximum value and minimum value and aratio a of a reflectance of a P-polarized light component to areflectance of an S-polarized light component on a surface of theeyeball E which surface corresponds to that pixel unit. Further, thediffusive reflection light component calculating section 18 determinesthe calculated diffusive reflection light component as a luminance valueof a unit region corresponding to that pixel unit. Thus, the imageprocessing device 10D can determine, as a luminance value of each ofunit regions, a diffusive reflection light component from which areflected light component has been at least partially removed.

FIG. 25 is graphs showing examples of effective radiance values ofpixels included in a pixel unit, estimated values of possible effectiveradiance values of the pixels included in the pixel unit, and estimatedvalues of a diffusion light component. (a) of FIG. 25 is a graph of apixel unit which reflected light enters in a case where light hasreached an eyeball E at an incident angle of 30° and has been reflectedby the eyeball E. (b) of FIG. 25 is a graph of a pixel unit whichreflected light enters in a case where light has reached an eyeball E atan incident angle of 20° and has been reflected by the eyeball E. (c) ofFIG. 25 is a graph of a pixel unit which reflected light enters in acase where light has reached an eyeball E at an incident angle of 10°and has been reflected by the eyeball E. In each of the graphs shown in(a) through (c) of FIG. 25, a horizontal axis represents a principalaxis direction of a polarizing element corresponding to each of pixelsincluded in a pixel unit, and a vertical axis represents an effectiveradiance value of a pixel. Moreover, in the example shown in FIG. 25, arefractive index of air is 1.0, and a refractive index of a cornea ofthe eyeball E is 1.376.

First, the following description discusses the pixel unit correspondingto the reflected light obtained from light whose incident angle to theeyeball E is 30°, as shown in (a) of FIG. 25. Here, in a case where theeffective radiance value is normalized as Ioutmax=1, Ioutmin is 0.60.Moreover, in the reflected light obtained from light whose incidentangle is 30°, a is 0.40.

In this case, the diffusive reflection light component Ik is calculatedas follows with use of the formula (6-1).

$\begin{matrix}{{Ik} = {\left( {{Ioutmin} - {a \times {Ioutmax}}} \right)/\left( {1 - a} \right)}} \\{= {\left( {0.60 - {0.40 \times 1.0}} \right)/\left( {1.0 - 0.40} \right)}} \\{\approx 0.33}\end{matrix}$

Next, the following description discusses the pixel unit correspondingto the reflected light obtained from light whose incident angle to theeyeball E is 20°, as shown in (b) of FIG. 25. Here, in a case where theeffective radiance value is normalized as Ioutmax=1, Ioutmin is 0.79.Moreover, in the reflected light obtained from light whose incidentangle is 20°, a is 0.688.

In this case, the diffusive reflection light component Ik is calculatedas follows with use of the formula (6-1).

$\begin{matrix}{{Ik} = {\left( {{Ioutmin} - {a \times {Ioutmax}}} \right)/\left( {1 - a} \right)}} \\{= {\left( {0.79 - {0.688 \times 1.0}} \right)/\left( {1.0 - 0.688} \right)}} \\{\approx 0.33}\end{matrix}$

Next, the following description discusses the pixel unit correspondingto the reflected light obtained from light whose incident angle to theeyeball E is 10°, as shown in (c) of FIG. 25. Here, in a case where theeffective radiance value is normalized as Ioutmax=1, Ioutmin is 0.94.Moreover, in the reflected light obtained from light whose incidentangle is 10°, a is 0.91.

In this case, the diffusive reflection light component Ik is calculatedas follows with use of the formula (6-1).

$\begin{matrix}{{Ik} = {\left( {{Ioutmin} - {a \times {Ioutmax}}} \right)/\left( {1 - a} \right)}} \\{= {\left( {0.94 - {0.91 \times 1.0}} \right)/\left( {1.0 - 0.91} \right)}} \\{\approx 0.33}\end{matrix}$

As such, in the example shown in FIG. 25, the diffusive reflection lightcomponent Ik is 0.33 in each of the cases where Ioutmax=1 and theincident angles of light to the eyeball E are 10°, 20°, and 30°. In anyof the angles, the value of Ik is substantially constant, and thereforethe above value of Ik seems to be correct.

[Embodiment 7]

The following description will discuss Embodiment 7 of the presentinvention. An image processing device of Embodiment 7 has aconfiguration similar to that of the image processing device 10D, exceptfor contents of a process carried out by the diffusive reflection lightcomponent calculating section 18, and therefore a block diagram and aflowchart are not illustrated.

The diffusive reflection light component calculating section 18 ofEmbodiment 7 calculates a diffusive reflection light component Ik foreach of pixel units by a formula (7-1) below.Ik=Ioutmin−Ip  (7-1)

The following description will discuss the formula (7-1).

In Embodiment 7, a first regular reflection light component Is has acomponent depending on a principal axis direction of a polarizingelement, whereas a second regular reflection light component Ip does notdepend on the principal axis direction of the polarizing element.Specifically, in Embodiment 7, the diffusive reflection light componentcalculating section 18 calculates Is by a formula (7-2) below.Is=Ioutmax−Ioutmin  (7-2)

Moreover, Ik is expressed by a formula (7-3) below.Ik=Ioutmax−(Is+Ip)  (7-3)

By modifying the formula (7-3) with use of the formula (7-2), a formula(7-4) below is obtained.Ik=Ioutmin−Ip  (7-4)

Ip is a value which can be calculated with use of the formula (6-7).Therefore, the diffusive reflection light component calculating section18 can calculate the diffusive reflection light component Ik by theformulae (7-4) and (6-7).

As above described, the diffusive reflection light component calculatingsection 18 of Embodiment 7 calculates two kinds of regular reflectionlight components and determines, as a luminance value of a unit regioncorresponding to a pixel unit, a value obtained by subtracting thecalculated regular reflection light component from a maximum value ofeffective radiance values of pixels included in the pixel unit. Even ina case where the diffusive reflection light component calculatingsection 18 executes such a process, the image processing device 10D candetermine, as a luminance value of each of unit regions, a diffusivereflection light component from which the regular reflection lightcomponent has been at least partially removed.

[Embodiment 8]

The following description will discuss Embodiment 8 of the presentinvention with reference to FIG. 26 and FIG. 27.

FIG. 26 is a block diagram illustrating a configuration of an electronicinformation apparatus 3 including an image processing device 10E inaccordance with Embodiment 8. As illustrated in FIG. 26, the electronicinformation apparatus 3 is different from the electronic informationapparatus 2 of Embodiment 7 in that the electronic information apparatus3 includes an image processing device 10E instead of the imageprocessing device 10D, and includes a polarized light irradiatingsection 40.

The polarized light irradiating section 40 irradiates a subject withpolarized light. For example, the polarized light irradiating section 40includes a light source such as an LED and a polarized light filterwhich causes only polarized light in a particular direction to passthrough.

The image processing device 10E is different from the image processingdevice 10D in that the image processing device 10E includes a reflectedimage presence/absence determining section 19. The reflected imagepresence/absence determining section 19 determines whether or not aregular reflection light component on a surface of the subject has anintensity that is equal to or less than a predetermined intensity. Inother words, the reflected image presence/absence determining section 19determines whether or not a reflected image occurs on the subject.

In a case where the reflected image presence/absence determining section19 has determined that no reflected image occurs on the subject, thepolarized light irradiating section 40 irradiates the subject withpolarized light. Moreover, in this case, a luminance value informationobtaining section 16 (luminance value information reobtaining section)obtains effective radiance values of the subject again while thepolarized light irradiating section 40 is irradiating the subject withpolarized light.

In Embodiment 8, the reflected image presence/absence determiningsection 19 calculates a ratio of a minimum value to a maximum value ofeffective radiance values for each of pixel units corresponding to theeyeball E. Further, the reflected image presence/absence determiningsection 19 calculates an average of the ratios in all the pixel unitscorresponding to the eyeball E and, (i) in a case where the average isequal to or more than a predetermined value, determines that noreflected image occurs and, (ii) in a case where the average is lessthan the predetermined value, determines that a reflected image occurs.

In a case where an incident angle of light to the subject is small, aratio of a minimum value to a maximum value becomes large amongeffective radiance values of pixels included in a pixel unitcorresponding to the incident angle. The predetermined value can be aratio of a minimum value to a maximum value among effective radiancevalues for a possible smallest incident angle to the subject.Specifically, the predetermined value can be, for example, 0.94. Thisvalue is a ratio of a minimum value to a maximum value among effectiveradiance values calculated by use of Fresnel's law for reflected lightobtained from light whose incident angle to the eyeball E is 10°.

FIG. 27 is a flowchart showing processes in the image processing device10E. As shown in FIG. 27, in the image processing device 10E, theluminance value information obtaining section 16 obtains effectiveradiance values of the subject (SC1), and then the reflected imagepresence/absence determining section 19 determines whether or not areflected image occurs (SD1, reflected image determining step). In acase where a reflected image occurs (Y in SD1), the image processingdevice 10E carries out processes of and subsequent to the step SC2, aswith the image processing device 10B.

On the other hand, in a case where no reflected image occurs (N in SD1),the luminance value information obtaining section 16 obtains effectiveradiance values of the subject again while the polarized lightirradiating section 40 is irradiating the subject with polarized light(SD2, luminance value information reobtaining step). In this case, theimage processing device 10E carries out processes of and subsequent tothe step SC2 with use of the effective radiance values of the subjectobtained in the step SD2.

In the image processing device 10E, in a case where no reflected imageoccurs on the eyeball E, the luminance value information obtainingsection 16 obtains effective radiance values of the subject includingthe eyeball E while the polarized light irradiating section 40 iscausing a reflected image. The diffusive reflection light componentcalculating section 18 (i) removes a regular reflection light componentwith use of a ratio a calculated by Fresnel's law from effectiveradiance values obtained by the luminance value information obtainingsection 16 and (ii) generates a picture image, and the authenticatingsection 15 carries out authentication. Therefore, in a case where theeyeball E is an imitation which is made of a substance different fromthat of a real eyeball, specifically, a substance having a refractiveindex that is different from the refractive index (n=1.376) of a cornea,authentication of a user is more likely to fail.

Note that the electronic information apparatus 3 does not necessarilyneed to include the polarized light irradiating section 40. For example,it is possible that another device, which is connected with theelectronic information apparatus 3 via wireless communication orwireline communication, includes the polarized light irradiating section40. Moreover, in the above described processes, the luminance valueinformation obtaining section 16 obtains effective radiance values ofthe subject in both the steps SC1 and SD2. However, a functional block(luminance value information reobtaining section) for obtainingeffective radiance values of the subject in the step SD2 can bedifferent from the luminance value information obtaining section 16.

Moreover, as above described, the image processing device 10E isdifferent from the image processing device 10D in that the imageprocessing device 10E includes the reflected image presence/absencedetermining section 19. In other words, the image processing device 10Ehas a configuration in which the reflected image presence/absencedetermining section 19 is added to the image processing device 10D.Note, however, that the image processing device of Embodiment 8 can havea configuration in which the reflected image presence/absencedetermining section 19 is added to another image processing device usingFresnel's law, e.g., the image processing device 10 or 10A.

[Embodiment 9]

The image processing device in accordance with an aspect of the presentinvention can carry out two or more processes among the above describedprocesses.

For example, an image processing device in accordance with an aspect ofthe present invention can execute one of the image processing describedin Embodiments 1, 2, and 4 through 7 and, in a case where authenticationfails, the image processing device can carry out authentication again byexecuting the executed image processing or another image processingother than the executed image processing. In a case where even thesecond authentication fails, the image processing device can obtain aneffective radiance distribution of the subject again or take a pictureimage of the subject again and, if necessary, further measure a distanceto the subject again. In a case where still another image processing issimultaneously or sequentially executed and authentication does notsucceed even by obtaining an effective radiance distribution of thesubject several times or taking a picture image of the subject severaltimes, the image processing device can carry out authentication based ona picture image generated by that still another image processing.

In the above described embodiments, in a case where authentication hasfailed, authentication is carried out again by changing a range of aniris detected by the iris detecting section 11. However, an imageprocessing device in accordance with an aspect of the present inventioncan be configured such that, in a case where authentication has failedonce or more, the luminance value information obtaining section 16obtains effective radiance values (takes a picture image) of the subjectagain. In a case where another image processing is simultaneously orsequentially executed and authentication does not succeed even byobtaining an effective radiance distribution of the subject severaltimes, the image processing device can carry out authentication based ona picture image generated by that another image processing.

An image processing device in accordance with an aspect of the presentinvention can simultaneously execute image processing(s) of otherembodiment(s) (of one or more of Embodiments 1, 2, 4, 5, and 7) with theimage processing of Embodiment 6.

FIG. 29 is a flowchart showing a flow of processes in the imageprocessing device which simultaneously executes the image processing ofEmbodiment 6 and another image processing. In the processes in FIG. 29,the processes of the steps SC1, SC5, and SC6 are identical with thosedescribed in the above embodiments, and are therefore not described herein detail.

In the processes shown in FIG. 29, the image processing device obtainsluminance value information of the subject (SC1), and thensimultaneously carries out (i) generation of a first post-removalpicture image in which a luminance value of each of unit regions hasbeen determined by the image processing of Embodiment 6 (SD1) and (ii)generation of a second post-removal picture image in which a luminancevalue of each of unit regions has been determined by the imageprocessing of another embodiment (SD2). After that, the image processingdevice determines whether or not a luminance value is equal to or morethan a predetermined threshold, for each of the unit regions in thefirst post-removal picture image (SD3). In a case where the luminancevalue is equal to or more than the threshold (Y in SD3), the luminancevalue of the unit region is not changed. On the other hand, in a casewhere the luminance value is not equal to or more than the threshold (Nin SD3), the luminance value of the unit region is changed to aluminance value of a unit region at a corresponding position in thesecond post-removal picture image. The image processing device executesthe determination of the step SD3 for all the unit regions, and executesthe process of the step SD4 in accordance with the determination result,and then carries out authentication with use of the obtained pictureimage (SC5). Further, the image processing device determines whether ornot authentication has succeeded (SC6). In a case where authenticationhas succeeded (Y in SC6), the image processing device ends the process.In a case where authentication has failed (N in SC6), the processreturns to the step SD1 again, and a first post-removal picture image isgenerated again. Specifically, in a case of N in the step SC6, the imageprocessing device executes processes of and subsequent to the step SC11shown in FIG. 24. Note that, in a case of N in the step SC6, it ispossible that a first post-removal picture image and a secondpost-removal picture image are both generated again by returning to boththe steps SD1 and SD2, instead of generating only a first post-removalpicture image again by returning to only the step SD1.

As above described, in the image processing of Embodiment 6, there is apossibility that a diffusive reflection light component disappears in aunit region in which an incident angle of light to the eyeball is small.In view of this, the image processing device determines a luminancevalue of each of unit regions by the image processing of Embodiment 6and then determines whether or not the luminance value of each of unitregions is equal to or more than the predetermined threshold. In a unitregion in which a luminance value is equal to or more than thethreshold, the luminance value is directly employed. On the other hand,in a unit region where a luminance value is not equal to or more thanthe threshold, a luminance value obtained by the image processing of theanother embodiment is employed as a luminance value of the unit region.With the configuration, the image processing device can prevent adiffusive reflection light component from disappearing.

[Additional Remarks]

Normally, the term “specular reflection light” is used only for visiblelight that is emitted in a direction toward a viewpoint (e.g., an eye ofa user, an image pickup device, or the like). However, in thisspecification, the term “specular reflection light” is used as a termrepresenting a concept including infrared light, in addition to visiblelight emitted in the direction toward a viewpoint.

In the above Embodiments 1 through 3, the image processing device 10 andthe like are mounted on the personal digital assistant 1 and the like.However, each of the image processing device 10 and the like of thoseembodiments can be mounted on the electronic information apparatus 2 orthe like instead of the personal digital assistant 1 or the like, aswith the image processing device 10B and the like of other embodiments.Each of the electronic information apparatuses 2 and 3 can be anintercom or a door of an automobile, which intercom or door has afunction to carry out authentication of a user, instead of the abovedescribed smart phone.

Moreover, in the above described Embodiments 1 through 3, theS-polarized light calculating section 12 identifies an incident angle oflight to the eyeball E which angle corresponds to a pixel, with use of adistance from the lens of the camera 20 to the surface of the eyeball Emeasured by the distance measuring device 30. However, for example, itis possible that the S-polarized light calculating section 12 calculatesthe distance based on a size of a picture image of the eyeball E in astate in which the eyeball E is in focus, and identifies an incidentangle of light to the eyeball E, which angle corresponds to a pixel,with use of the distance. Alternatively, for example, it is possiblethat the S-polarized light calculating section 12 assumes in advance adistance from the lens of the camera 20 to the surface of the eyeball E(e.g., in a case where the personal digital assistant 1 is a smartphone, 20 cm to 30 cm), and identifies an incident angle of light to theeyeball E, which angle corresponds to a pixel, with use of the assumeddistance. In this case, it is possible that a plurality of distances areassumed, an incident angle of light to the eyeball E, which anglecorresponds to a pixel, is identified with use of those distances, and areflected image is removed. Moreover, in this case, the personal digitalassistant 1 does not necessarily need to include the distance measuringdevice 30.

In the above descriptions with reference to FIG. 3, it is described thata reflected image is “removed”. Further, FIG. 3 itself includes thephrase “S-wave/P-wave removal”. However, in an aspect of the presentinvention, the image processing device does not necessarily need tocompletely remove a reflected image, as long as a regular reflectionlight component which forms the reflected image is at least partiallyremoved, i.e., the reflected image is reduced.

FIG. 28 is a view showing an arrangement of pixel units in the camera20. (a) of FIG. 28 is a view illustrating a state in which pixel unitseach of which is made up of nine pixels are arranged so as not tooverlap with each other. (b) of FIG. 28 is a view illustrating a statein which pixel units each of which is made up of four pixels arearranged so as not to overlap with each other. (c) of FIG. 28 is a viewillustrating a state in which pixel units each of which is made up ofnine pixels are arranged so as to partially overlap with each other. (d)of FIG. 28 is a view illustrating a state in which pixel units each ofwhich is made up of four pixels are arranged so as to partially overlapwith each other. In (a) through (d) of FIG. 28, a region surrounded bydotted lines, a region surrounded by dashed lines, and a regionsurrounded by dashed dotted lines are separate pixel units.

In the above described embodiments, in the camera 20, one (1) pixel isincluded in only one (1) pixel unit. In other words, as illustrated in(a) and (b) of FIG. 28, pixel units are arranged so as not to overlapwith each other. However, in an aspect of the present invention, it ispossible that one (1) pixel is included in two or more pixel units. Thatis, as illustrated in (c) and (d) of FIG. 28, pixel units can bearranged so as to partially overlap with each other.

In a case where pixel units partially overlap with each other, thenumber of pixel units is larger than a case where pixel units do notoverlap with each other. Therefore, in a case where pixel unitspartially overlap with each other, it is possible to expect that aregular reflection light component can be removed more.

As the number of principal axis directions of polarizing elementscorresponding to pixels included in one (1) pixel unit increases, it ispossible to remove a regular reflection light component moreappropriately. However, at least one pixel corresponding to one (1)principal axis direction needs to be prepared and therefore, as thenumber of principal axis directions increases, the number of pixelsincluded in one (1) pixel unit accordingly increases.

In a case where the number of pixels included in one (1) pixel unitincreases and pixel units are arranged so as not to overlap with eachother as illustrated in (a) and (b) of FIG. 28, there is a problem thata resolution of a picture image decreases. Practically, it is preferablethat the number of principal axis directions of polarizing elementscorresponding to pixels included in one (1) pixel unit is two or moreand nine or less.

Moreover, in the above described embodiments, each of the imageprocessing devices 10 and 10A through 10E includes the iris detectingsection 11 which identifies an iris region. However, each of the imageprocessing devices 10 and 10A through 10E can include acornea/iris/pupil detecting section for identifying a cornea, iris, orpupil region of a user, instead of the iris detecting section 11. Aprocess of identifying the cornea, iris, or pupil region by thecornea/iris/pupil detecting section is similar to the process ofdetecting an iris region by the iris detecting section 11, and is knownin the field of iris authentication and the like. The cornea/iris/pupildetecting section transmits position information of pixels correspondingto the identified cornea, iris, or pupil region to the minimum luminancevalue selecting section 17, the minimum luminance value estimatingsection 17A, or the diffusive reflection light component calculatingsection 18, and to the authenticating section 15 and the like, dependingon embodiments. Processes of other blocks included in each of the imageprocessing devices 10 and 10A through 10E are similar to the processesdescribed in the above embodiments.

The following description will briefly discuss an example of a processof detecting an iris region (or cornea, iris, or pupil region) by theiris detecting section 11 (or cornea/iris/pupil detecting section).First, the iris detecting section 11 (or cornea/iris/pupil detectingsection) carries out sharpening, edge detection, and binarization withrespect to effective radiance values obtained in pixels of the imagepickup device. In the edge detection, for example, a sobel filter can beused. Moreover, in binarization, for example, it is possible to use amoving average method or a partial image dividing method. The irisdetecting section 11 (or cornea/iris/pupil detecting section) carriesout Hough transform with respect to the binarized effective radiancevalues, and detects a circular region as an iris (or cornea, iris, orpupil region).

Moreover, in the above described embodiments, the radius r is a radiusof the eyeball E. However, the radius r can be a curvature radius of acornea.

The following description will discuss a case where the subject inEmbodiment 1 includes eyeballs E of both eyes. In this case, numericalvalues are defined as follows.

-   -   Rd: Distance between (i) intermediate point on line segment        connecting centers of two eyeballs E and (ii) lens of camera 20    -   R1: Distance between center of one eyeball E and lens of camera        20    -   L: Distance between center of one eyeball E and the intermediate        point

Here, Rd is a known value measured by the distance measuring device 30,and R1 cannot be directly measured by the distance measuring device 30.A method of calculating L will be described later. Note that Rd does notnecessarily need to be measured by the distance measuring device 30, andcan be calculated by a calculating method using only the camera 20, aswith R described in Embodiment 6.

FIG. 30 is a view for explaining a case where the subject includeseyeballs E of both eyes. In this case, a value of R1 can be calculatedby a formula (10-1) below.R1=sqrt(Rd ²+(L/2)²)  (10-1)

Here, sqrt(Rd²+(L/2)²) represents a square root of Rd²+(L/2)². Bysubstituting R1 for R in the above described formulae (1-1) and (1-2)and calculating R1 by the formula (10-1), the image processing device 10of Embodiment 1 can carry out a process in a case where the subject iseyeballs E of both eyes.

The following description will discuss a method for calculating L. Forthe calculation of L, numerical values are defined as follows.

-   -   α: Angle formed by center and end part of cornea with respect to        center of eyeball E    -   H1: Number of pixels over radius of cornea of one eyeball E in        picture image    -   H2: Number of pixels over distance between centers of corneas of        both eyeballs E in picture image

Among those values, a can be, for example, 35° in view of a generalratio of cornea accounting for a circle having a radius r. Note that avalue of a can be another value, e.g., an arbitrary value falling withina range of 20° or more and 50° or less. Moreover, H1 and H2 can beobtained from a picture image.

In this case, a value of L can be calculated by a formula (10-2) below.L=H2×(r×sin α)/H1  (10-2)

By substituting the value of L calculated by the formula (10-2) for theformula (10-1), it is possible to calculate a value of R1.

In the above described embodiments, a refractive index of a cornea isn=1.376. However, the refractive index of the cornea is not limited tothe above described example and can be, for example, 1.335, 1.337,1.3375, 1.37, or 1.38, or another value.

[Example Realized with Use of Software]

Each of the image processing devices 10 and 10A through 10E can berealized by a logic circuit (hardware) provided in an integrated circuit(IC chip) or the like or can be alternatively realized by software asexecuted by a central processing unit (CPU).

In the latter case, each of the image processing devices 10 and 10Athrough 10E includes: a CPU that executes instructions of a program thatis software realizing the foregoing functions; read only memory (ROM) ora storage device (each referred to as “storage medium”) storing theprogram and various kinds of data in such a form that they are readableby a computer (or a CPU); and random access memory (RAM) into which theprogram is loaded in executable form. An object of an aspect of thepresent invention can be achieved by a computer (or a CPU) reading andexecuting the program stored in the storage medium. The storage mediummay be “a non-transitory tangible medium” such as a tape, a disk, acard, a semiconductor memory, and a programmable logic circuit. Further,the program may be supplied to or made available to the computer via anytransmission medium (such as a communication network and a broadcastwave) which enables transmission of the program. Note that an aspect ofthe present invention can also be achieved in the form of a computerdata signal in which the program is embodied via electronic transmissionand which is embedded in a carrier wave.

[Main Points]

The image processing method in accordance with an aspect 1 of thepresent invention includes the steps of: obtaining a picture image of asubject (eyeball E) taken by an image pickup device (light-receivingelement 22) in which pixel units are two-dimensionally arranged, each ofthe pixel units including a plurality of pixels that are associated withrespective of a plurality of polarizing elements (21 a through 21 o)whose principal axis directions are different from each other (S1); andcalculating, with use of an output from the image pickup device, aluminance distribution of S-polarized light depending on an incidentangle (θ) with respect to the subject, the incident angle (θ) beingdetermined in accordance with a position on the subject which positioncorresponds to a two-dimensional position of the pixel units in theimage pickup device (SA1, SA2, SB1).

According to the method, for example, in a case where the subject is aneyeball, an incident angle with respect to the eyeball when a pictureimage of the eyeball is taken is determined in accordance with aposition on the eyeball. That is, assuming that there are a firstvirtual line (straight line L1 shown in FIG. 6) which connects aposition on the eyeball with a center of the eyeball and a secondvirtual line (straight line L2 shown in FIG. 6) which connects theposition on the eyeball with a center of a lens of a camera included inthe image pickup device, an angle formed between the first virtual lineand the second virtual line is an incident angle with respect to theeyeball.

Meanwhile, in the image pickup device, the pixel units each made up of aplurality of pixels associated with the respective plurality ofpolarizing elements whose principal axis directions are different fromeach other are two-dimensionally arranged. A position on the eyeball, anincident angle with respect to the eyeball at the position, and atwo-dimensional position of a pixel unit in the image pickup device areassociated with each other. Moreover, the pixel unit is associated withthe plurality of polarizing elements whose principal axis directions aredifferent from each other, and therefore outputs from the plurality ofpixels constituting the pixel unit vary depending on a distributionstate, on the eyeball, of polarized light included in reflected lightfrom the eyeball. In particular, the outputs from the plurality ofpixels associated with the respective plurality of polarizing elementswhose principal axis directions are different from each other reflect aluminance of S-polarized light. That is, the output from the imagepickup device reflects the distribution state, on the eyeball, ofS-polarized light included in reflected light from the eyeball.Therefore, it is possible to calculate, with use of an output of theimage pickup device, a luminance distribution of S-polarized lightdepending on an incident angle with respect to the eyeball.

The luminance distribution of S-polarized light thus obtained variesbetween an eyeball of a living body and an imitation of an eyeball. Fromthis, for example, in a case where a picture image of an imitation of aneyeball is taken as a subject, a luminance distribution of S-polarizedlight of the imitation is different from a distribution of an eyeball,and it is therefore possible to determine that the subject is not aneyeball. As such, the image processing method in accordance with theaspect 1 can be used, for example, to determine whether the subject isan eyeball or not.

Moreover, the luminance distribution of S-polarized light calculated asabove can be used, for example, to remove an image of an objectreflected on an eyeball due to influence of outside light.

Therefore, according to the image processing method in accordance withthe aspect 1, it is possible to improve accuracy in iris authentication.

The image processing method in accordance with an aspect 2 of thepresent invention preferably further includes, in the aspect 1, the stepof subtracting the luminance distribution of S-polarized light from aluminance distribution of the picture image.

In a case where another object such as a landscape or a person isreflected on the eyeball in the method, a part of reflected light fromthe eyeball which part corresponds to the another object is mainlyspecular reflection light, and the specular reflection light includesthe S-polarized light. Therefore, according to the method, it ispossible to reduce an unnecessary noise image reflected on the eyeball,by subtracting the luminance distribution of S-polarized light from aluminance distribution of the picture image of the eyeball.

The image processing method in accordance with an aspect 3 of thepresent invention preferably further includes, in the aspect 1 or 2, thestep of calculating, from the luminance distribution of S-polarizedlight, a luminance distribution of P-polarized light depending on anincident angle with respect to the subject based on Fresnel's law.

In the method, the specular reflection light from the eyeball in a casewhere another object is reflected on the eyeball often includesP-polarized light, as well as S-polarized light. A luminancedistribution of P-polarized light also varies between an eyeball of aliving body and an imitation of an eyeball. Therefore, according to themethod, for example, it is possible to increase accuracy in determiningwhether the subject is an eyeball or not.

The image processing method in accordance with an aspect 4 of thepresent invention preferably further includes, in the aspect 3, the stepof subtracting, from the luminance distribution of the picture image,the luminance distribution of S-polarized light and the luminancedistribution of P-polarized light.

According to the method, it is possible to further reduce an unnecessarynoise image reflected on the eyeball, by subtracting the luminancedistribution of S-polarized light and the luminance distribution ofP-polarized light from the luminance distribution of the picture imageof the eyeball.

In the image processing method in accordance with an aspect 5 of thepresent invention, it is preferable in any of the aspects 1 through 4that, in the step of calculating the luminance distribution ofS-polarized light, in a case where the incident angle is a Brewsterangle, a pixel unit corresponding to the Brewster angle is identified, aluminance value of S-polarized light with respect to the Brewster angleis calculated by subtracting a minimum value from a maximum value ofluminance values of pixels included in the pixel unit which has beenidentified, and a luminance value of S-polarized light with respect toan incident angle other than the Brewster angle is calculated inaccordance with Fresnel's law, on the basis of the luminance value ofS-polarized light with respect to the Brewster angle.

In the method, the following is known: that is, with respect to theBrewster angle, specular reflection light hardly includes P-polarizedlight and mainly includes S-polarized light. Therefore, outputs (i.e.,luminance values) from a plurality of pixels constituting the pixel unitcorresponding to the Brewster angle reflect a change in transmittance ofS-polarized light with respect to the plurality of polarizing elementswhose principal axis directions are different from each other. Fromthis, it is possible to calculate the luminance value of S-polarizedlight with respect to the Brewster angle by subtracting a minimum valuefrom a maximum value of those luminance values.

In an eyeball, a function representing a luminance distribution ofS-polarized light with respect to an incident angle accords withFresnel's law and is known. Therefore, it is possible to calculate aluminance value of S-polarized light with respect to an incident angleother than the Brewster angle based on the theory, by calculating theluminance value of S-polarized light with respect to the Brewster angle.

In the image processing method in accordance with an aspect 6 of thepresent invention, it is preferable in any of the aspects 1 through 5that the step of calculating a luminance value of S-polarized light bysubtracting a minimum value from a maximum value of luminance values ofpixels included in the pixel unit is repeated for each of the pluralityof pixel units included in the image pickup device, and the luminancedistribution of S-polarized light is calculated by associatingcalculated luminance values of S-polarized light with incident anglescorresponding to the respective plurality of pixel units.

According to the method, as already described, it is possible tocalculate a luminance value of S-polarized light by subtracting aminimum value from a maximum value of luminance values of pixelsincluded in a pixel unit. By repeating the step for each of theplurality of pixel units included in the image pickup device andassociating the calculated luminance values of S-polarized light withthe respective incident angles corresponding to the plurality of pixelunits, it is possible to calculate the luminance distribution of theS-polarized light based on actually measured values.

Note that the method in accordance with this aspect in which theluminance distribution of S-polarized light is calculated based onactually measured values can be carried out in a case where the methodof the above described aspects in which the luminance distribution ofS-polarized light is calculated based on the theory do not work well.

The image processing device (10) in accordance with an aspect 7 of thepresent invention includes: an S-polarized light calculating section(12) which calculates, with use of an output from an image pickup device(light-receiving element 22), a luminance distribution of S-polarizedlight depending on an incident angle with respect to a subject (eyeballE) in a picture image of the subject taken by the image pickup device inwhich pixel units are two-dimensionally arranged, each of the pixelunits including a plurality of pixels that are associated withrespective of a plurality of polarizing elements whose principal axisdirections are different from each other, and the incident angle beingdetermined in accordance with a position on the subject which positioncorresponds to a two-dimensional position of any of the pixel units.

According to the configuration, it is possible to bring about an effectsimilar to that of the aspect 1.

The image processing program in accordance with an aspect 8 of thepresent invention is an image processing program for causing a computerto function as the image processing device in accordance with the aspect7, the image processing program causing the computer to function as theS-polarized light calculating section.

The image processing device in accordance with the aspects of thepresent invention may be realized by a computer. In this case, an imageprocessing program which realizes the image processing device with useof a computer by causing the computer to function as sections (softwareelements) of the image processing device, and a computer-readablestorage medium in which the image processing program is stored, are alsoencompassed in the aspects of the present invention.

The aspects of the present invention are not limited to the embodiments,but can be altered by a skilled person in the art within the scope ofthe claims. An embodiment derived from a proper combination of technicalmeans each disclosed in a different embodiment is also encompassed inthe technical scope of the aspects of the present invention. Further, itis possible to form a new technical feature by combining the technicalmeans disclosed in the respective embodiments.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Nonprovisional application claims priority on Patent ApplicationNo. 2016-091832 filed in Japan on Apr. 28, 2016 and on PatentApplication No. 2017-074535 filed in Japan on Apr. 4, 2017, the entirecontents of which are hereby incorporated by reference.

REFERENCE SIGNS LIST

10, 10A, 10B, 10C, 10D, 10E: Image processing device

15: Authenticating section

16: Luminance value information obtaining section

17: Minimum luminance value selecting section (image generating section)

17A: Minimum luminance value estimating section (image generatingsection)

18: Diffusive reflection light component calculating section (imagegenerating section)

19: Reflected image presence/absence determining section

40: Polarized light irradiating section

The invention claimed is:
 1. An image processing method comprising: aluminance value information obtaining step of obtaining effectiveradiance values of a subject by an image pickup device in which pixelunits are two-dimensionally arranged, each of the pixel units includinga plurality of pixels which are associated with respective of aplurality of polarizing elements whose principal axis directions aredifferent from each other, and each of the effective radiance valuesbeing an effective radiance value in the image pickup device; an imagegenerating step of generating a picture image including an image of thesubject with use of the effective radiance values obtained from thesubject, in the image generating step, a luminance value being obtainedby at least partially removing a regular reflection light component on asurface of the subject from the effective radiance values of theplurality of pixels included in each of the pixel units corresponding toat least part of the subject, and the picture image being generated as aset of unit regions each of which has the luminance value; and wherein:in the image generating step, a minimum value of possible effectiveradiance values in each of the pixel units is estimated based oneffective radiance values of the plurality of pixels included in thatpixel unit, and a luminance value of each of the unit regions isdetermined based on the minimum value.
 2. The image processing method asset forth in claim 1, wherein: in the image generating step, the minimumvalue is estimated by applying a trigonometric function or a polynomialexpression, in which an angle of a principal axis direction of each ofthe plurality of polarizing elements is a variable, to the effectiveradiance values of the plurality of pixels included in each of the pixelunits.
 3. The image processing method as set forth in claim 1, wherein:in the image generating step, a maximum value and a minimum value ofeffective radiance values in each of the pixel units is identified basedon effective radiance values of the plurality of pixels included in thatpixel unit, a diffusive reflection light component in each of the pixelunits is calculated by using the maximum value and the minimum value anda ratio of a reflectance of a P-polarized light component to areflectance of an S-polarized light component, the ratio being a ratioon a surface of the subject which surface corresponds to the pixel unitsand being calculated based on Fresnel's law, and the diffusivereflection light component is determined as a luminance value of each ofthe unit regions.
 4. The image processing method as set forth in claim1, wherein: in the image generating step, a maximum value and a minimumvalue of effective radiance values in each of the pixel units isidentified based on effective radiance values of the plurality of pixelsincluded in that pixel unit, a first regular reflection light componentis calculated by subtracting the minimum value from the maximum value, asecond regular reflection light component is calculated by using thefirst regular reflection light component and a ratio of a reflectance ofa P-polarized light component to a reflectance of an S-polarized lightcomponent, the ratio being a ratio on a surface of the subject whichsurface corresponds to the pixel units and being calculated based onFresnel's law, and a value obtained by subtracting the first regularreflection light component and the second regular reflection lightcomponent from the maximum value is determined as a luminance value ofeach of the unit regions.
 5. The image processing method as set forth inclaim 1, further comprising: a reflected image determining step ofdetermining whether or not a regular reflection light component on asurface of the subject, which component is included in the effectiveradiance values obtained in the luminance value information obtainingstep, has an intensity equal to or less than a predetermined intensity;and a luminance value information reobtaining step of obtainingeffective radiance values while the subject is being irradiated withpolarized light in a case where, in the reflected image determiningstep, the intensity of the regular reflection light component on thesurface of the subject has been determined to be equal to or less thanthe predetermined intensity, in a case where the luminance valueinformation reobtaining step has been executed, a regular reflectionlight component, which is on the surface of the subject and is includedin the effective radiance values obtained in the luminance valueinformation reobtaining step, being at least partially removed in theimage generating step.
 6. An image processing device, comprising: animage pickup device in which pixel units are two-dimensionally arranged,each of the pixel units including a plurality of pixels which areassociated with respective of a plurality of polarizing elements whoseprincipal axis directions are different from each other; a luminancevalue information obtaining section which obtains effective radiancevalues of a subject by the image pickup device, each of the effectiveradiance values being an effective radiance value in the image pickupdevice; an image generating section which generates a picture imageincluding an image of the subject with use of the effective radiancevalues obtained from the subject, the image generating section obtaininga luminance value by at least partially removing a regular reflectionlight component on a surface of the subject from effective radiancevalues of the plurality of pixels included in each of the pixel unitscorresponding to at least part of the subject, and generates the pictureimage as a set of unit regions each of which has the luminance value;and wherein: the image generating section estimates a minimum value ofpossible effective radiance values in each of the pixel units based oneffective radiance values of the plurality of pixels included in thatpixel unit; and the image generating section determines a luminancevalue of each of the unit regions based on the minimum value.
 7. Theimage processing device as set forth in claim 6, wherein: the imagegenerating section estimates the minimum value by applying atrigonometric function or a polynomial expression, in which an angle ofa principal axis direction of each of the plurality of polarizingelements is a variable, to the effective radiance values of theplurality of pixels included in each of the pixel units.
 8. The imageprocessing device as set forth in claim 6, wherein: the image generatingsection identifies a maximum value and a minimum value of effectiveradiance values in each of the pixel units based on effective radiancevalues of the plurality of pixels included in that pixel unit; the imagegenerating section calculates a diffusive reflection light component ineach of the pixel units by using the maximum value and the minimum valueand a ratio of a reflectance of a P-polarized light component to areflectance of an S-polarized light component, the ratio being a ratioon a surface of the subject which surface corresponds to the pixel unitsand being calculated based on Fresnel's law; and the image generatingsection determines the diffusive reflection light component as aluminance value of each of the unit regions.
 9. The image processingdevice as set forth in claim 6, wherein: the image generating sectionidentifies a maximum value and a minimum value of effective radiancevalues in each of the pixel units based on effective radiance values ofthe plurality of pixels included in that pixel unit; the imagegenerating section calculates a first regular reflection light componentby subtracting the minimum value from the maximum value; the imagegenerating section calculates a second regular reflection lightcomponent by using the first regular reflection light component and aratio of a reflectance of a P-polarized light component to a reflectanceof an S-polarized light component, the ratio being a ratio on a surfaceof the subject which surface corresponds to the pixel units and beingcalculated based on Fresnel's law; and the image generating sectiondetermines a value, which is obtained by subtracting the first regularreflection light component and the second regular reflection lightcomponent from the maximum value, as a luminance value of each of theunit regions.
 10. The image processing device as set forth in claim 6,further comprising: a reflected image determining section whichdetermines whether or not a regular reflection light component on asurface of the subject, which component is included in the effectiveradiance values obtained by the luminance value information obtainingsection, has an intensity equal to or less than a predeterminedintensity; and a luminance value information reobtaining section whichobtains effective radiance values while the subject is being irradiatedwith polarized light in a case where the reflected image determiningsection has determined that the intensity of the regular reflectionlight component on the surface of the subject is equal to or less thanthe predetermined intensity, in a case where the luminance valueinformation reobtaining section has obtained the effective radiancevalues of the subject, the image generating section at least partiallyremoves a regular reflection light component which is on the surface ofthe subject and is included in the effective radiance values obtained bythe luminance value information reobtaining section.
 11. The imageprocessing device as set forth in claim 6, wherein: a part of thesubject is an iris; and said image processing device further comprisesan authenticating section which carries out authentication of a userbased on the picture image generated by the image generating section.